Ask Reid Lyon – Reading First Implementation

May 19, 2008

(Guest post by Reid Lyon)

“The implementation of Reading First has been a hot topic, especially since the release of the recent impact study. Has the implementation of the program been successful?  What were the most important issues that arose regarding the implementation of Reading First? What lessons have been learned?”

 

With respect to whether the implementation of the program has been successful, the short answer is it depends.  Amber Winkler, the research director for the Fordham Foundation, recently reported in the Gadfly that Reading First is “perhaps the best-implemented education program in federal history.”  This may be true in some states, but not in others.  Dr. Winkler highlights Georgia, Oregon, and California as states that got the implementation process right.

 

On the other hand, Texas and several other states made successful implementation unbearably difficult.  In Texas, for example, a process was initially in place to award districts Reading First funding if their proposals were rated above a particular score by the grant review group.  Proposals from districts scoring below a threshold value were to be resubmitted following technical assistance to equip the district with in depth understanding of Scientifically Based Reading Research (SBRR) and to incorporate those concepts in their resubmission along with assessment and accountability requirements.  For some reason, a decision was made to award all districts Reading First funding irrespective of the quality of their applications.  Moreover, eligible districts received Reading First funding before any technical assistance was in place and before any baseline reading assessments could be administered for program evaluation purposes.  Not good.

 

But there is a great deal more involved in the implementation of a complex program beyond ensuring the quality of grant applications, and few states and districts had their hands around all of the essential conditions that must be in place to embed and bring to scale an initiative as intricate as Reading First.  At first blush, it would seem that the existence of a converging body of evidence relevant to reading development, reading difficulties, and reading instruction would have facilitated implementation fidelity.  But a substantial research base, bipartisan political support, and hefty funding only go so far.  The devil is in the details, and strategies for implementation at the federal level and in many states either were not appraised of the details or simply felt, as in Texas, that the money had to flow immediately no matter what, reflecting education’s love affair with entitlement programs.

 

One detail that gets right in your face immediately when you are implementing a program as complex as Reading First is that you have to manage coordinated systems change at federal, state, district, school and classroom levels.  Complexity theorists like W.L. Miller and his team use the metaphor of a jazz band when discussing how individuals within an effective system perform their own tasks in concert with others in achieving a desired goal.  Each contributor is responsible for certain tasks, but always listening to the other members of the orchestra to determine how their own actions contribute to the whole. This is tough to do when the band members come from different generations and different musical perspectives.  Getting into the groove takes a good deal of practice and a willingness to expand one’s thinking.  These are not features that characterize public education.   Basically, any evaluation of a program like Reading First must drill down into how this coordination played out and how long it took.  

 

Another common-sense detail that was not planned for in the implementation schedule was the fact that many hard-working folks in schools and classrooms who have gotten used to doing things in a certain way were being asked to change their routines and to try something new. In some cases this meant a decision was made to stop certain programs and replace them with others that many were unfamiliar with. When this occurs, teachers and leaders must have confidence that what is being implemented provides advantages to students over existing practices – and, in many instances, the case for this was not made prior to implementation.  While this may seem a bit fluffy, it is important to understand that public-school educators are under a constant barrage of new magic bullets, fads, and aggressive textbook company representatives all selling materials “based on SBRR.”  Implementation experts will tell you that without teacher and leader buy-in, any program, no matter how effective, will not realize its potential.  Fortunately, the majority of those leading and teaching in Reading First districts and schools saw the clear advantages offered students by the program as they observed poor readers become good readers.  But this took a while. 

 

In my interactions with many Reading First programs over the past six years, I did notice some common conditions that were in place when implementation fidelity was strong.  In addition to the details noted above, strong implementers embraced data and accountability for results.  States like Alabama integrated robust professional development with continuous coaching and feedback for both teachers and leaders.  Instructional programs were selected not only on the basis of their alignment with SBRR but because they were practical, useful, and beneficial to students.  Teachers were treated as self-determined professionals and responded by taking ownership and responsibility for their parts in the jazz band.  And building-level leadership ensured that teachers and coaches had the necessary time to plan, review student data, and collaborate in differentiating instruction for individual students based on their performance data.

 

The lessons learned are many, but I can think of three big ones.  Let’s start with the way Congress and the feds typically expect complex programs to be in place, in full operation, as soon legislation is passed.  Reading First embodied so many new concepts and requirements that, in my view, the first year should have been spent in providing technical assistance and professional development to states and districts even prior to the submission of Reading First grants.  I can’t tell you the number of times I saw the thousand-yard stare following my mention of SBRR, progress monitoring, data-driven instruction, or comprehensive reading programs.  We are talking significant mismatch between the requirements of the legislation and the background knowledge of many grant applicants; not to mention that the grants were competitive – a novel concept in education formula funding.

 

And if you were on the ground during the first two years of the program, this is what you would typically observe:  Teachers were first learning to understand, administer and use the results of assessments to inform instruction.  As they were learning these new concepts, they were also taking part in state reading academies to learn more about  the foundation of SBRR (in 5 areas of reading in k-1, in 4 areas  of reading in 2-3). In addition, as they were learning and using new assessments and taking part in professional development academies and workshops, they were simultaneously learning how to use a new approach to instruction and how to integrate core program instruction with additional interventions when required to meet individual student needs. This was done at the same time they were learning about center activities, grouping students for instruction and aligning and using supported classroom libraries.

 

It is important to ask whether any program that has added this amount of new learning to a teacher’s other responsibilities – including going to IEP meetings, attending parent conferences, preparing for their instruction in math, social studies and science, serving on school-wide committees and a host of other tasks – could demonstrate substantial gains after three years.  Give me a break.  What is amazing is that despite this unbelievable load, Reading First coordinators, teachers and their leaders rose to the occasion and have done and are doing a superb job.

 

Lesson Number One: Take a year to develop the infrastructure essential for program implementation. 

 

Lesson Number Two: During this first year,  make sure that all involved at every level understand the essential conditions that have to be in place to coordinate and implement a massive and unique program and to anticipate the need to customize some of its features based on individual district and school characteristics.

 

Lesson Number Three – and this is for the Department of Education: The next time Congress gives you $25 million dollars a year for six years to carry out an ongoing evaluation of a program, for God’s sake design and implement the evaluation commensurate with the initiation of the program.  This was no time to carry out a delayed and abbreviated evaluation when the complexity and uniqueness of a program demanded comprehensive, continuous and systematic feedback to ensure improvements in implementation where needed.

Leave a Comment » | instructional reform, politics | Tagged: , , | Permalink
Posted by Greg Forster

Ask Reid Lyon

May 13, 2008

(Guest Post by Reid Lyon)

“How did scientific research become influential in guiding federal education ‎policy given the field’s historical reliance on ideology, untested ‎assumptions, anecdotes, and superstition to inform both policy and practice?‎”

It has not been an easy journey. In fact it’s like getting a root canal every other week.  What makes it tough is that you are always bumping up against the anti-scientific thinking that has had a misguided influence on the perceived value of research throughout the history of education and increasingly in the past two decades. Many researchers have tried to infuse scientific research into education policy over the years but it never gained political traction. Jeanne Chall gave her career to this cause, but the political will was never there.  Many at the policy level rarely listened to her, much less took her advice.  Chall would tell me frequently that by not basing reading instruction on research we do grave harm to the students education seeks to serve. I repeated her wisdom every time I testified before congressional committees. I also repeated myself time and again that education like other sectors that serve the public, must have reliable information about what works, why it works, and how it works. The alternative was to basically throw mud against a wall and see what sticks – a practice in place for a very long time.  I would argue that scientific research and dissemination of reliable information to the educational community is non-negotiable given that all sectors of a productive society depend on an educated workforce. To be sure, many in the education community sure got medieval on me for holding to this position.

But logic, congressional testimony, research syntheses, or policy papers were not going to change the culture in education which had reinforced an “everything and anything goes” spirit for the past century. Infusing research into policy and practice was going to take strong  support from  a senior  member or members of congress who could argue the need in a compelling way.  Bill Goodling, past chair of the House Education and the Workforce Committee, did just that and in 1996 began to support the concept of “research-based education”.  Goodling was a past educator and was floored when he began to delve into the fact that millions of kids could not read.  His staff learned that the NIH had been studying reading development and reading difficulties since 1965 so they called me in early 1996 to brief the chairman on what we knew about reading from a research standpoint.  At that time, I directed the NICHD Reading Research program at the NIH. During the briefing, he was literally taken aback to learn that NICHD/NIH had studied over 40,000 good and not so good readers, many of them over time, and we had a good idea of what it took to learn to read and what to do about reading difficulties. He could not understand why there was such a massive gap between what research had demonstrated vis-à-vis reading development and instruction and what was actually taught to teachers and  implemented in schools. 

1996 turned out to be a pretty important year in bringing the massive reading failure issue before the public and mobilizing some scientific efforts. It was also an important year for laying the foundation for research-based education policy as it is reflected in federal legislation today.  President Clinton called attention to the tragedy of reading failure in his State of the Union address that year.  His attention to the issue clearly put the problem on congressional radar screens.  In the same year, the Department of Education and the NICHD supported the convening of a National Research Council (NRC) panel to synthesize and summarize research on the prevention of reading difficulties.  Interestingly, at the same time, state leaders were becoming interested in the “research to policy and practice issue”.  Interestingly, in 1996, then Texas governor George Bush asked me and members of several strong research teams in Texas and around the country to brief him on how scientific research in reading could help reduce reading failure in Texas. In one of the meetings he asked a pretty prescient question about how scientific research could help kids whose first language was Spanish to learn to listen, speak, read, and write in English.  This question actually gave birth to the NICHD national “Spanish to English” study carried out in multiple sites across the country.

 But during that year it was Goodling and his staff who went to work on the specifics and the need to educate other congressional members not only about the drastic need to address the reading issue, but to emphasize the role of scientific research in solving educational problems.  He and his staff devoted substantial time in 1996 reviewing the NICHD reading research. In early 1997, he and his counterparts in the senate held hearings on literacy development and the role of scientific research in developing and implementing effective instructional practices. It came as a surprise to me that in my testimony that year before both House and Senate committees, members asked about research on reading and how it could help guide policy and practice.  Their interest in using scientific research to guide  practice and policies would later extend to other education programs beyond reading as I was asked to cover the issue in testimony on Title I, Head Start, and IDEA re-authorizations which took place over the next 9 years.  And Goodling was the first legislator to formally infuse scientific research in reading into a federal education program.  In 1998, He sponsored the Reading Excellence Act, which for the first time required that federal funding be contingent on states and local districts using scientifically based programs.

To further underscore the interest and commitment that congress had in using research to guide federal education policy, Senator Thad Cochran and Representative Anne Northup asked the NICHD in 1998 to convene a National Reading Panel (NRP) to build on the findings of the 1996 NRC panel on preventing reading difficulties in young children.  The NRP was tasked to undertake a review of research on reading instruction that would identify the types of programs and principles that were most effective in improving reading proficiency.  While the NRC and NRP reports were initiated and published during the Clinton administration, the Bush administration used the findings not only to craft Reading First but to serve as an example of the overarching principle that educational policy and instructional practices should be predicated on research.  From this principal evolved the established of the Institute of Educational Sciences, the NRC Report on “Scientific Research in Education”, the Partnership for Reading which served as a resources to disseminate scientific research findings, and the What Works Clearing House.  Private groups such as the Council for Excellence in Government, which established the Coalition for Evidence Based Policy,  began to contribute to this effort as well.

If you take all of this together, the recent influx of educational science into policy came about through a concerted effort to solve a national reading problem. Using research to guide educational policy and program development has now been extended far beyond reading.   A number of actions such as congressional hearings, funding of research reports on science in education, requiring federal funds be contingent on the use of research-based programs and approaches, passing legislation such as the Education Sciences Reform Act of 2002, and building a federal infrastructure which, by its inclusion of the Institute of Educational Sciences and the What Works Clearing House, explicitly sent the message that research-based policies and programs were the rule, not the exception.  It is the case that much of the integration of actions and events was strategic and designed to provide a role for scientific research in education.  A research to policy and practice culture had to be strengthened through federal legislation and in the scientific infrastructure within the Department of Education. 

Time will tell if the gains made in using research to guide education policy will last.  History tells us that education is impatient and subject to fads, superstition, anecdotes, and the next magic bullet.  To be sure, education is more political than scientific and subject to all the negatives that the political world brings but few of the positives. And many do not understand that by its cannons, evidence is apolitical.  There is a tendency to forget that research is not only essential for informing policy but critical for improving policies and programs once in place. But trial and error has become a habit in education and it will take real courage and persistence to overcome that.  In a sense, the world of education policy is  like a slinky–it can expand to take new steps, but it ultimately recoils back to its original configuration.   All this said, I am optimistic.

12 Comments | instructional reform, methodology, politics | Tagged: , , | Permalink
Posted by Jay P. Greene

Reading First — There’s More to the Story

May 8, 2008

(Guest Post By Reid Lyon)

I have received many calls from people attending the International Reading Association’s conference in Atlanta informing me that many of the members are celebrating the null findings presented in the Reading First Impact Study Interim Report.  But you could have predicted that behavior easily from past behavior.  While IRA as an organization has been a supporter of Reading First, many celebrating have wanted the program to fail from day one.  But you have to wonder whether the detractors have read the actual interim report or just the press accounts of the evaluation to date?   The press coverage overwhelmingly reported the null findings without coverage of the limitations of the study – limitations that should be considered as much as the findings themselves- particularly when drawing any conclusions from the data.  There has been little mention of the degree of overlap between Reading First and non-reading First schools and no mention – at least that I can find –of the fact that little time has elapsed since Reading First has been implemented which makes it very difficult to draw conclusions at this time.   I am hopeful that these issues will be addressed in detail in the final report given that this information can provide more guidance for improvement.   However, in my view and in the whole scheme of things, implementing a program as complex as Reading First will take a bit more time than three years or less.

Indeed, it has been more the rule than the exception that during the first two years of Reading First implementation in districts and schools, teachers were first learning to understand, administer and use the results of assessments to inform instruction.

To jack up the complexity, as they were learning these new concepts, they were also taking part in state reading academies to learn more about  the foundations of SBRR (in 5 areas of reading in k-1, in 4 areas  of reading in 2-3). To make implementation even more complex, as they were learning and using new assessments and taking part in professional development academies and workshops, they were simultaneously learning how to use new approaches to instruction and how to integrate core program instruction with additional interventions when required to meet individual student needs. This was done at the same time they were learning about center activities, grouping students for instruction and aligning and using supported classroom libraries.

It is important to ask whether any program that has added this amount of new learning to a teacher’s other responsibilities including going to IEP meetings, attending parent conferences, preparing for their instruction in math, social studies and science, serving on school wide committees and a host of other tasks could demonstrate substantial gains after only two years. What is amazing is that despite this unbelievable load, Reading First teachers and their leaders rose to the occasion and have done and are doing a superb job. Also note that the GAO and OMB reports show that they feel that this job is essential and that it is having a major impact.

To be sure, as one of the individuals involved in conceptualizing and drafting the Reading First legislation, it is a no brainer that I am passionate about its potential.  That said, the data must speak for the effectiveness of the program.  If we wanted to avoid using effectiveness data to monitor and improve the program, we would not have mandated one of the most comprehensive evaluations that have ever been applied to any educational program.   The fact that the evaluation plan as written in the legislation was not carried out will require some explanation at some point.  More importantly, it is incumbent on both supporters and opponents of Reading First to pay very close attention to the details in the interim evaluation.  It is impossible to make informed decisions about improving a program, gutting a program, or reducing the funding for a program on the basis of ambiguous findings.  And folks will have to read the details themselves – apparently they will not learn about them from the press.

3 Comments | instructional reform | Tagged: , , | Permalink
Posted by Jay P. Greene

Reading First Disappoints

May 1, 2008

A new study released today by the U.S. Department of Education finds that Reading First, the phonics based federally-funded reading program, failed to yield improvements in reading comprehension.  The study doesn’t demonstrate that it is ineffective to emphasize phonics; only that it was ineffective to adopt this particular program.  That could be because there are problems with the design or implementation of Reading First.  Or it could be because the control group was also receiving phonics instruction without the particular elements of Reading First.  And it is possible that phonics is less effective than some people thought.

Whatever the explanation, this well-designed study undermines confidence that instructional reform, like Reading First, can alone transform the educational system

16 Comments | instructional reform, research reports | Tagged: , , , | Permalink
Posted by Jay P. Greene

%d bloggers like this: