(Guest Post by Reid Lyon)
“How did scientific research become influential in guiding federal education policy given the field’s historical reliance on ideology, untested assumptions, anecdotes, and superstition to inform both policy and practice?”
It has not been an easy journey. In fact it’s like getting a root canal every other week. What makes it tough is that you are always bumping up against the anti-scientific thinking that has had a misguided influence on the perceived value of research throughout the history of education and increasingly in the past two decades. Many researchers have tried to infuse scientific research into education policy over the years but it never gained political traction. Jeanne Chall gave her career to this cause, but the political will was never there. Many at the policy level rarely listened to her, much less took her advice. Chall would tell me frequently that by not basing reading instruction on research we do grave harm to the students education seeks to serve. I repeated her wisdom every time I testified before congressional committees. I also repeated myself time and again that education like other sectors that serve the public, must have reliable information about what works, why it works, and how it works. The alternative was to basically throw mud against a wall and see what sticks – a practice in place for a very long time. I would argue that scientific research and dissemination of reliable information to the educational community is non-negotiable given that all sectors of a productive society depend on an educated workforce. To be sure, many in the education community sure got medieval on me for holding to this position.
But logic, congressional testimony, research syntheses, or policy papers were not going to change the culture in education which had reinforced an “everything and anything goes” spirit for the past century. Infusing research into policy and practice was going to take strong support from a senior member or members of congress who could argue the need in a compelling way. Bill Goodling, past chair of the House Education and the Workforce Committee, did just that and in 1996 began to support the concept of “research-based education”. Goodling was a past educator and was floored when he began to delve into the fact that millions of kids could not read. His staff learned that the NIH had been studying reading development and reading difficulties since 1965 so they called me in early 1996 to brief the chairman on what we knew about reading from a research standpoint. At that time, I directed the NICHD Reading Research program at the NIH. During the briefing, he was literally taken aback to learn that NICHD/NIH had studied over 40,000 good and not so good readers, many of them over time, and we had a good idea of what it took to learn to read and what to do about reading difficulties. He could not understand why there was such a massive gap between what research had demonstrated vis-à-vis reading development and instruction and what was actually taught to teachers and implemented in schools.
1996 turned out to be a pretty important year in bringing the massive reading failure issue before the public and mobilizing some scientific efforts. It was also an important year for laying the foundation for research-based education policy as it is reflected in federal legislation today. President Clinton called attention to the tragedy of reading failure in his State of the Union address that year. His attention to the issue clearly put the problem on congressional radar screens. In the same year, the Department of Education and the NICHD supported the convening of a National Research Council (NRC) panel to synthesize and summarize research on the prevention of reading difficulties. Interestingly, at the same time, state leaders were becoming interested in the “research to policy and practice issue”. Interestingly, in 1996, then Texas governor George Bush asked me and members of several strong research teams in Texas and around the country to brief him on how scientific research in reading could help reduce reading failure in Texas. In one of the meetings he asked a pretty prescient question about how scientific research could help kids whose first language was Spanish to learn to listen, speak, read, and write in English. This question actually gave birth to the NICHD national “Spanish to English” study carried out in multiple sites across the country.
But during that year it was Goodling and his staff who went to work on the specifics and the need to educate other congressional members not only about the drastic need to address the reading issue, but to emphasize the role of scientific research in solving educational problems. He and his staff devoted substantial time in 1996 reviewing the NICHD reading research. In early 1997, he and his counterparts in the senate held hearings on literacy development and the role of scientific research in developing and implementing effective instructional practices. It came as a surprise to me that in my testimony that year before both House and Senate committees, members asked about research on reading and how it could help guide policy and practice. Their interest in using scientific research to guide practice and policies would later extend to other education programs beyond reading as I was asked to cover the issue in testimony on Title I, Head Start, and IDEA re-authorizations which took place over the next 9 years. And Goodling was the first legislator to formally infuse scientific research in reading into a federal education program. In 1998, He sponsored the Reading Excellence Act, which for the first time required that federal funding be contingent on states and local districts using scientifically based programs.
To further underscore the interest and commitment that congress had in using research to guide federal education policy, Senator Thad Cochran and Representative Anne Northup asked the NICHD in 1998 to convene a National Reading Panel (NRP) to build on the findings of the 1996 NRC panel on preventing reading difficulties in young children. The NRP was tasked to undertake a review of research on reading instruction that would identify the types of programs and principles that were most effective in improving reading proficiency. While the NRC and NRP reports were initiated and published during the Clinton administration, the Bush administration used the findings not only to craft Reading First but to serve as an example of the overarching principle that educational policy and instructional practices should be predicated on research. From this principal evolved the established of the Institute of Educational Sciences, the NRC Report on “Scientific Research in Education”, the Partnership for Reading which served as a resources to disseminate scientific research findings, and the What Works Clearing House. Private groups such as the Council for Excellence in Government, which established the Coalition for Evidence Based Policy, began to contribute to this effort as well.
If you take all of this together, the recent influx of educational science into policy came about through a concerted effort to solve a national reading problem. Using research to guide educational policy and program development has now been extended far beyond reading. A number of actions such as congressional hearings, funding of research reports on science in education, requiring federal funds be contingent on the use of research-based programs and approaches, passing legislation such as the Education Sciences Reform Act of 2002, and building a federal infrastructure which, by its inclusion of the Institute of Educational Sciences and the What Works Clearing House, explicitly sent the message that research-based policies and programs were the rule, not the exception. It is the case that much of the integration of actions and events was strategic and designed to provide a role for scientific research in education. A research to policy and practice culture had to be strengthened through federal legislation and in the scientific infrastructure within the Department of Education.
Time will tell if the gains made in using research to guide education policy will last. History tells us that education is impatient and subject to fads, superstition, anecdotes, and the next magic bullet. To be sure, education is more political than scientific and subject to all the negatives that the political world brings but few of the positives. And many do not understand that by its cannons, evidence is apolitical. There is a tendency to forget that research is not only essential for informing policy but critical for improving policies and programs once in place. But trial and error has become a habit in education and it will take real courage and persistence to overcome that. In a sense, the world of education policy is like a slinky–it can expand to take new steps, but it ultimately recoils back to its original configuration. All this said, I am optimistic.
Jay P. Greene's Blog 
See also Mike Petrilli’s post on Reading First today at Flypaper http://www.edexcellence.net/flypaper/index.php/2008/05/reading-first-miracle-worker/
Dr. Lyon describes the alternative to science as throwing mud against a wall and seeing what sticks. I think even that is giving too much credit to the anti-science crowd. “Seeing what sticks” is the job of science. The alternative to science is to keep throwing mud against the wall year after year without bothering to find out what results.
Thanks for this informative post – having spent my career so far on the governance side of the education field, I didn’t know most of this history on the pedagogical side. It really is sad that we had all this knoweldge about reading for all that time and few even knew about it! Not surprising, of course, but sad.
Your post, especially your acknowledgement of Jeanne Chall and others who did this work before anyone outside the field was paying attention, reminds me of something J.S. Mill wrote in his defense of free speech. He said there were twenty Martin Luthers before the Reformation. What he meant was that Luther’s ideas weren’t new; lots of people had said the same things before. What changed was the social conditions that made it possible for him to survive attempts to silence him and to reach a wide audience with those ideas. Obviously this isn’t quite the same thing, but the dynamic feels similar.
I’m wondering Dr. Lyon regrets the decision to permit the vast majority of Reading First funding to go to untested and scientifically unvalidated reading programs instead of to the reading programs that had actual scientific support. That decision was a gross misuse of scientific research. It was a logical fallacy to conclude that since all the scientifically validated reading programs contain that five essential components of reading instruction that any program containing these components would also be effective. Now it seems that the chickens are coming home to roost with the interim study, flawed though it may be, showing that many of these untested reading programs showed little increased reading achievement. I would not be surprised if the fallout of this bad decision sets scientifically based reading instruction back by years.
I certainly do regret the decision to permit the vast majority of Reading First funding to be allocated for untested programs. However, it was neither I nor Bob Sweet – the two individual who drafted the legislation – that made this decision. In the drafts we submitted for congressional review, funding was contingent on a program’s demonstration of effectiveness using the appropriate methods and designs required to determine causal effects. We also coupled this language with a sentence requiring that claims of effectiveness had to be combined with information indicating g the conditions under which effectiveness was established.
This language was diluted at the congressional level by members and staff who felt that restricting funding to only those programs that had established effectiveness would make implementation very difficult. Neither Sweet nor I were involved in that decision. As I have written many times, I did not agree then and I do not agree now with the scrapping of our language and replacing it with the much broader “based on” language. I am on the record many times stating that this congressional decision allowed for a significant opportunity for many publishers to hawk their wares under the guise of being “based on” SBRR. This went against the entire thrust and spirit of Reading First. What you might want to do is research this issue more deeply and look at the number of publishers and vendors that lobbied congress with vigor to change our original language.
I should have also pointed out in my response to Mr. DeRosa that the Reading First Impact Study as presented in the interim report DID NOT address the requirement in the law that the evaluation would analyze the degree to which different instructional materials improved reading proficiency – See section 1205 – in the Reading First legislation. I am sure there will be more written on the considerable difference between the impact evaluation that was carried out by IES and the comprehensive evaluation that was required in the legislation. I discussed this issue in addition to the dilution of the Reading First effectiveness language in a recent interview published in EdNews.org
I didn’t mean to imply that it as your or Bob Sweet’s decision to alter the statutory language. At this point there does not appear to be any independent source that confirms how this change came about.
There is your 1/6/06 interview with Nancy Salvato in which you state “The Department of Education made the decision to make the criteria more general. ” Then there is your 3/27/2007 correction of that interview in which you state essentially what you’ve stated above. Then there are the the many statements post IG reports by you and Bob Sweet maintaining this position.
So, given that there was some question as to where the funding line was being enforced by DoE in the IG report, the GAO report, Congressman Miller and Senator Kennedy, I think it would be helpful to nail down how this chane ecame into being and by whom as a defense to these accusations that are relying on the fact that no independent source is backing you and Sweet up and has admitted responsibility for this change to date.
Any thoughts?
, and I am aware of all your statements you both have made.
There does seem to be some confusion as to how that language came to be changedand at whose insistence.
Typo alert : those last two paragraphs should have been deleted.
I’d also point out that the inclusion of all these untested programs undoubtedly affected the impact study results and is even more of a reason why the programs funded under RF should be evaluated separately in the final report — to avoid a loosening of the statutory language in the future if we expect better results out of SBR legislation going forward.
The OIG has a record of the original language drafted by Bob Sweet and myself on the Reading First effectiveness criteria. The same language was sent to the OIG when they requested recommendations for re-authorization. I had incorrectly assumed that the Department had modified the language during their drafting of the Reading First guidance which would not have involved either Sweet or me. My incorrect assumption led to my incorrect answer to one of Salvato’s questions. Had I asked Bob, he would have told me what he has reported – the language went through several iterations in the negotiations process with precise wording being debated every step of the way. Neither Sweet nor I would have any final say in the structure of the language passed into law. I would contact Bob directly as HE is the record having done the heavy lifting on turning research principles into legislative language and guiding the legislation through the conference negotiations. He can certainly provide a very accurate summary of the inner workings of that process. What I do know is that the opposition to the language as submitted was bipartisan. The republicans, in general, felt it was too prescriptive. Many democrats had program favorites that would not have met the effectiveness criteria at that time. There are clearly overlaps between these two groups. Members on both sides were also lobbied heavily by publishers and vendors.
The problem is that we will not know from the impact study, at least as it was recently summarized, whether particular educational materials showed any particular effect on reading proficiency. This question was not included in the current evaluation design f as required in the law (see Section 1205) that we drafted. The evaluation was narrowed to the four questions indicated in the interim report despite having substantial funding to carry out the most comprehensive of evaluations. The null effect reported in the interim study is difficult to interpret because of sampling and implementation issues, even before you get to program specific effectiveness. Questions regarding effectiveness have to be addressed within the context of fidelity of implementation factors, the qualifications of those imparting a particular program, the additional reading activities taking place in school and elsewhere, the time allotted for reading instruction, the issue of combining a core program with a supplemental intervention, and several other potential confounds. Even if a program has been found to be effective in trials elsewhere, that program might not achieve the same effectiveness in a different set of contextual variables. That is why the original language also included a statement of the conditions under which a program had been found to be effective. These are not trivial issues when considering program specific impacts. There is substantial complexity involved from a design and methodology perspective and obviously from an implementation perspective. It can be done but it is expensive. That is why $25 million dollars a year for five years was allocated to the Reading First evaluation.
So it sounds like those in Congress close to educatation would have known the precise scope of the language and which kinds of reading programs were intended to be funded and which were not based on the final language. They would have also known that DOE for all its careless procedures actually funded the right programs and there is no evidence that they funded any program improperly despite the potential conflicts. So why the political theater?
I think Mr. DeRosa gives a little too much credit to the depth of knowledge that many congressional members and staffers bring with them when working on education legislation. Believe me, SBRR was as foreign a concept to many in congress as it was to states, districts, and schools. That is not a criticism. They are inundated with a tremendous amount of other legislative information on a myriad of issues. Also, much of the exposure to the characteristics of scientifically based research and its promise in guiding policy that members would receive at congressional hearings would typically be presented by researchers who would overwhelm members by using technical language and would frequently qualify every point made. As I mentioned in an earlier post, Bill Goodling and his staff understood in depth the need to guide education policy and practices based on the best evidence possible, as did a few other members on both sides of the aisle.
But people much more experienced than I working in the bowels of the congressional process can explain the impact of historical ideas about the federal role in education, the very strong influence of outside special interest groups and lobbyists, and the political infighting that is conducted just to make others look like they have failed on what finally emerges from conference as law. To me it’s a real ugly process. I heard more times than I can count that I just did not understand how it is done, and I probably still do not get it. For example, recall that the majority of education formula funding grants was considered by recipients to be, in reality, entitlement awards. A recipient basically had to account for process, not product, and let the feds know where the money ended up. Outcome was a concept that did not resonate with many in the education world. The prescriptive nature of Reading First was foreign to many legislators, and many fought hard to loosen the criteria every step of the way, not only on effectiveness criteria, but on other critical aspects of the law (accountability guidelines, data reporting requirements, etc.).
What I learned is that the best of intentions go into a legislative meat grinder and come out packaged in law that gives you some of what was intended and takes away some of what was intended. I learned that logic was not high on the list of essentials in the legislative process. Given this, my partner Bob Sweet would counsel me to keep in mind we were inching forward and it simply had no chance of being perfect. We were able to bring the concept of research-based education into the law of the land. That had never been done before. And that was precisely why many in the education community fought and are still fighting to remove the word “research-based” from the legislative lexicon. And, to be sure, they are receiving help from mistakes made in the implementation of Reading First and from wildly misconstrued findings reported in the interim impact study report. But it is the fact changes in the legislative language were not a result of some conspiracy to fill the coffers of particular publishers or vendors. They came about through a compromise process driven by many special interests particularly a strong opposition to create federal rules that states should follow if they were going to receive federal money. Let’s pile up that money on the stump and let the feeding frenzy begin.
children disabilities…
What I do say, is that I don’ t want to be turned into something I’ m not, and I don’ t wish that upon anyone, including this child. The aim of support or treatment for this child, or me, or anyone, should not be to“ normalize” them. When in …
kDeRosa,I was looking for information about you because I noticed how you were so committed to pushing Dr.Lyon up against the wall.
Upon doing some searching I saw you were a fan of Dr.Allington`s ideology.
That`s like attacking Obama when you`re a Republican,pretty pathetic don`t you think?
As a committed educator I`ve used the NICHD research study for years and I`ve helped so many children and many more to come!
FYI You`re beating up the wrong guy!
Obviously,you care about being right not about improving the odds for children,tough world out there if you can`t read!
Curious about which publisher put you up to your dog attack,very vicious.Job well done!