The final report of the U.S. Department of Education commissioned evaluation of Reading First was released yesterday. According to Russ Whitehurst, the USDOE’s director of the Institute for Education Sciences: “I don’t think anyone should be celebrating that the federal government has spent $6 billion on a reading program that has had no impact on reading comprehension.”
(Guest post by Reid Lyon)
Jay’s analysis of Response to Instruction, also known as Response to Intervention (RTI), and the need for additional reforms to help ensure effectiveness is provocative and informative. I agree with a good deal of his analysis but feel the need to expand the discussion a bit.
First off, his piece offers a wake-up call to those who are prone to fall in love with magic bullets that will close the achievement gap and reduce referrals to special education. One of my dreams is to provide policymakers and educational leaders two free tattoos to be placed anywhere they want on their persons that read: (1) “Necessary but not sufficient,” and (2) “Great policy idea, but implementation is a bitch.” Jay’s points address both tattoos.
RTI is a noble and well researched concept. One of its major goals is to reduce referrals to special education by documenting that the student’s learning difficulties are not because of inadequate instruction in general education classrooms but because of a disability. Years ago, S. Allen Cohen provided us with a more interesting term for lousy teaching which he called “dyspedagogia” (I believe this was tongue in cheek).
But the fact is most kids identified for special education and labeled as having a Learning Disability (LD) are not LD but achieve poorly because of “dyspedagogia”. In fact, our research over the past 20 years has taught us that scientifically based early reading intervention provided through a tiered approach to instruction can reduce the percentage of LD from upwards of 22% to between 2% and 10% in some states and LEAs. This is a very good thing given that LD referrals and placements constitute about 50% of all referrals to special education, and reading disabilities comprise about 80% of kids identified with LD.
Those working on the development of RTI knew that classroom teachers and special educators are from different planets, with entirely different backgrounds in curriculum and instruction and entirely different professional languages. This makes collaboration and the provision of coherent and systematic instruction impossible. In essence, RTI was developed, in part, to underscore the need to develop a common assessment and instructional language between well intentioned teachers with different specializations so that instruction was not so fragmented, and so kids would not feel like ping-pong balls when receiving entirely different types of instruction on the same day from classroom teachers and special educators. There was no other alternative – typical colleges of education would not provide the professional development necessary to accomplish the level of shared training and collaboration skills essential for effective and differentiated instruction.
Related to this, RTI as a concept was also developed to take advantage of the converging research that prevention through early identification and intervention provides a more effective approach to reducing academic failure than hanging out waiting until the kid chucks his books through the window in the third or fourth grade. RTI procedures can differ across LEAs but they typically screen all students in kindergarten or early first grade to determine which students require enhanced instruction in the classroom or more intensive instruction in smaller groups, while still under the direction of the regular classroom teacher. If the kid demonstrates little academic (or behavioral) growth following more intense instruction, then special education becomes a possibility. It is critical to remember that being placed in special education is also a tiered process. The questions are: Does the student have a disability? (RTI contributes to answering this question); and, if yes, does the student require special education? (RTI contributes to answering this question).
So we have two elements in RTI that we know can increase achievement (at least in reading): (1) collaboration/common language and (2) prevention is where it’s at. These elements are inseparable. But we know that these two elements cannot work the way they are supposed to unless the folks who run the system know all of the potential barriers to the implementation of RTI and ensure that essential conditions to support the initiative are in place. There are examples of districts implementing successful RTI initiatives that increase students’ reading achievement and reduce referrals to special education. There are other examples where the initiative has not resulted in changes in achievement or referral outcomes.
So what is going on? It will be important to figure out what works in some districts and not others. Susan Hall has recently published a very user-friendly book that lays out what districts and schools did in implementing RTI in a way that resulted in substantial reductions in referrals and significant increases in reading achievement: A Principal’s Guide: Implementing Response to Intervention. I am not hawking this book, as there are others that lay out the conditions that are essential for implementing RTI and scaling it (see Dianne Haager et al. for specific evidence of effectiveness).
The bottom line is that effective implementers of RTI have had to do a tremendous amount of study and planning to ensure that the initiative actually makes a difference. Jay lays out some the barriers, including problems with implementing a program that actually takes funding away from your district or schools, and persuading educators to replace programs and procedures that they have used for some time.
But there are others as well. For example, how do you overcome the fact that intervention services in schools are often funded by separate entitlement programs, especially Title I and IDEA, that have specific eligibility criteria that make it difficult to co-mingle funds to support school-wide programs? How do you implement programs that have been typically isolated from general education? And how can school leaders and teachers avoid the mistakes that result in limited or no effectiveness, not to mention that the excitement for change and increased morale will be crushed?
If you did a factor analysis of all the crap that can derail the implementation of RTI, these are the most common errors:
–Focusing Too Many Resources on Administering and Collecting Assessment Data Rather Than Ensuring that Staff Use the Data to Inform Instruction
-Viewing Purchased Programs as Silver Bullets Rather Than Aides to Help Well Prepared Teachers Make Informed Instructional Decisions
-Confusing Awareness Training with Implementation Training
-Using Ineffective Practices to Train Teachers
-Underestimating the Magnitude of Change
-Taking on Too Many Grade Levels and Schools the First year
-Beginning the Implementation Without a Comprehensive Implementation Plan
-Failure to View the Implementation as a Systems-Wide Change
The good news is that districts and schools that have effective RTI programs in place know they can’t make the mistakes above and provide incentives that have trumped the traditional financial rewards that have potential for increasing referral rates.
I am sure that I have taken up too much space with details that may be of little interest to policymakers. But the details are what make RTI effective, and when RTI works, it really works. However, if you can’t deal with complexity, either-or concrete thinking, or have an allergic reaction to human and systems change, you might as well blow off trying to implement RTI.
Jay has done a service laying out the big picture issues. Implementing additional reforms to increase the probability that RTI can succeed is essential, as he has articulated. But the return on investment is only as good as understanding and addressing the amount of grunt work involved. It’s that “necessary but not sufficient” thing. I know this is self-evident, but we sure keep trying to do one without the other.
(Guest post by Reid Lyon)
“The implementation of Reading First has been a hot topic, especially since the release of the recent impact study. Has the implementation of the program been successful? What were the most important issues that arose regarding the implementation of Reading First? What lessons have been learned?”
With respect to whether the implementation of the program has been successful, the short answer is it depends. Amber Winkler, the research director for the Fordham Foundation, recently reported in the Gadfly that Reading First is “perhaps the best-implemented education program in federal history.” This may be true in some states, but not in others. Dr. Winkler highlights Georgia, Oregon, and California as states that got the implementation process right.
On the other hand, Texas and several other states made successful implementation unbearably difficult. In Texas, for example, a process was initially in place to award districts Reading First funding if their proposals were rated above a particular score by the grant review group. Proposals from districts scoring below a threshold value were to be resubmitted following technical assistance to equip the district with in depth understanding of Scientifically Based Reading Research (SBRR) and to incorporate those concepts in their resubmission along with assessment and accountability requirements. For some reason, a decision was made to award all districts Reading First funding irrespective of the quality of their applications. Moreover, eligible districts received Reading First funding before any technical assistance was in place and before any baseline reading assessments could be administered for program evaluation purposes. Not good.
But there is a great deal more involved in the implementation of a complex program beyond ensuring the quality of grant applications, and few states and districts had their hands around all of the essential conditions that must be in place to embed and bring to scale an initiative as intricate as Reading First. At first blush, it would seem that the existence of a converging body of evidence relevant to reading development, reading difficulties, and reading instruction would have facilitated implementation fidelity. But a substantial research base, bipartisan political support, and hefty funding only go so far. The devil is in the details, and strategies for implementation at the federal level and in many states either were not appraised of the details or simply felt, as in Texas, that the money had to flow immediately no matter what, reflecting education’s love affair with entitlement programs.
One detail that gets right in your face immediately when you are implementing a program as complex as Reading First is that you have to manage coordinated systems change at federal, state, district, school and classroom levels. Complexity theorists like W.L. Miller and his team use the metaphor of a jazz band when discussing how individuals within an effective system perform their own tasks in concert with others in achieving a desired goal. Each contributor is responsible for certain tasks, but always listening to the other members of the orchestra to determine how their own actions contribute to the whole. This is tough to do when the band members come from different generations and different musical perspectives. Getting into the groove takes a good deal of practice and a willingness to expand one’s thinking. These are not features that characterize public education. Basically, any evaluation of a program like Reading First must drill down into how this coordination played out and how long it took.
Another common-sense detail that was not planned for in the implementation schedule was the fact that many hard-working folks in schools and classrooms who have gotten used to doing things in a certain way were being asked to change their routines and to try something new. In some cases this meant a decision was made to stop certain programs and replace them with others that many were unfamiliar with. When this occurs, teachers and leaders must have confidence that what is being implemented provides advantages to students over existing practices – and, in many instances, the case for this was not made prior to implementation. While this may seem a bit fluffy, it is important to understand that public-school educators are under a constant barrage of new magic bullets, fads, and aggressive textbook company representatives all selling materials “based on SBRR.” Implementation experts will tell you that without teacher and leader buy-in, any program, no matter how effective, will not realize its potential. Fortunately, the majority of those leading and teaching in Reading First districts and schools saw the clear advantages offered students by the program as they observed poor readers become good readers. But this took a while.
In my interactions with many Reading First programs over the past six years, I did notice some common conditions that were in place when implementation fidelity was strong. In addition to the details noted above, strong implementers embraced data and accountability for results. States like Alabama integrated robust professional development with continuous coaching and feedback for both teachers and leaders. Instructional programs were selected not only on the basis of their alignment with SBRR but because they were practical, useful, and beneficial to students. Teachers were treated as self-determined professionals and responded by taking ownership and responsibility for their parts in the jazz band. And building-level leadership ensured that teachers and coaches had the necessary time to plan, review student data, and collaborate in differentiating instruction for individual students based on their performance data.
The lessons learned are many, but I can think of three big ones. Let’s start with the way Congress and the feds typically expect complex programs to be in place, in full operation, as soon legislation is passed. Reading First embodied so many new concepts and requirements that, in my view, the first year should have been spent in providing technical assistance and professional development to states and districts even prior to the submission of Reading First grants. I can’t tell you the number of times I saw the thousand-yard stare following my mention of SBRR, progress monitoring, data-driven instruction, or comprehensive reading programs. We are talking significant mismatch between the requirements of the legislation and the background knowledge of many grant applicants; not to mention that the grants were competitive – a novel concept in education formula funding.
And if you were on the ground during the first two years of the program, this is what you would typically observe: Teachers were first learning to understand, administer and use the results of assessments to inform instruction. As they were learning these new concepts, they were also taking part in state reading academies to learn more about the foundation of SBRR (in 5 areas of reading in k-1, in 4 areas of reading in 2-3). In addition, as they were learning and using new assessments and taking part in professional development academies and workshops, they were simultaneously learning how to use a new approach to instruction and how to integrate core program instruction with additional interventions when required to meet individual student needs. This was done at the same time they were learning about center activities, grouping students for instruction and aligning and using supported classroom libraries.
It is important to ask whether any program that has added this amount of new learning to a teacher’s other responsibilities – including going to IEP meetings, attending parent conferences, preparing for their instruction in math, social studies and science, serving on school-wide committees and a host of other tasks – could demonstrate substantial gains after three years. Give me a break. What is amazing is that despite this unbelievable load, Reading First coordinators, teachers and their leaders rose to the occasion and have done and are doing a superb job.
Lesson Number One: Take a year to develop the infrastructure essential for program implementation.
Lesson Number Two: During this first year, make sure that all involved at every level understand the essential conditions that have to be in place to coordinate and implement a massive and unique program and to anticipate the need to customize some of its features based on individual district and school characteristics.
Lesson Number Three – and this is for the Department of Education: The next time Congress gives you $25 million dollars a year for six years to carry out an ongoing evaluation of a program, for God’s sake design and implement the evaluation commensurate with the initiation of the program. This was no time to carry out a delayed and abbreviated evaluation when the complexity and uniqueness of a program demanded comprehensive, continuous and systematic feedback to ensure improvements in implementation where needed.
(Guest Post by Reid Lyon)
“How did scientific research become influential in guiding federal education policy given the field’s historical reliance on ideology, untested assumptions, anecdotes, and superstition to inform both policy and practice?”
It has not been an easy journey. In fact it’s like getting a root canal every other week. What makes it tough is that you are always bumping up against the anti-scientific thinking that has had a misguided influence on the perceived value of research throughout the history of education and increasingly in the past two decades. Many researchers have tried to infuse scientific research into education policy over the years but it never gained political traction. Jeanne Chall gave her career to this cause, but the political will was never there. Many at the policy level rarely listened to her, much less took her advice. Chall would tell me frequently that by not basing reading instruction on research we do grave harm to the students education seeks to serve. I repeated her wisdom every time I testified before congressional committees. I also repeated myself time and again that education like other sectors that serve the public, must have reliable information about what works, why it works, and how it works. The alternative was to basically throw mud against a wall and see what sticks – a practice in place for a very long time. I would argue that scientific research and dissemination of reliable information to the educational community is non-negotiable given that all sectors of a productive society depend on an educated workforce. To be sure, many in the education community sure got medieval on me for holding to this position.
But logic, congressional testimony, research syntheses, or policy papers were not going to change the culture in education which had reinforced an “everything and anything goes” spirit for the past century. Infusing research into policy and practice was going to take strong support from a senior member or members of congress who could argue the need in a compelling way. Bill Goodling, past chair of the House Education and the Workforce Committee, did just that and in 1996 began to support the concept of “research-based education”. Goodling was a past educator and was floored when he began to delve into the fact that millions of kids could not read. His staff learned that the NIH had been studying reading development and reading difficulties since 1965 so they called me in early 1996 to brief the chairman on what we knew about reading from a research standpoint. At that time, I directed the NICHD Reading Research program at the NIH. During the briefing, he was literally taken aback to learn that NICHD/NIH had studied over 40,000 good and not so good readers, many of them over time, and we had a good idea of what it took to learn to read and what to do about reading difficulties. He could not understand why there was such a massive gap between what research had demonstrated vis-à-vis reading development and instruction and what was actually taught to teachers and implemented in schools.
1996 turned out to be a pretty important year in bringing the massive reading failure issue before the public and mobilizing some scientific efforts. It was also an important year for laying the foundation for research-based education policy as it is reflected in federal legislation today. President Clinton called attention to the tragedy of reading failure in his State of the Union address that year. His attention to the issue clearly put the problem on congressional radar screens. In the same year, the Department of Education and the NICHD supported the convening of a National Research Council (NRC) panel to synthesize and summarize research on the prevention of reading difficulties. Interestingly, at the same time, state leaders were becoming interested in the “research to policy and practice issue”. Interestingly, in 1996, then Texas governor George Bush asked me and members of several strong research teams in Texas and around the country to brief him on how scientific research in reading could help reduce reading failure in Texas. In one of the meetings he asked a pretty prescient question about how scientific research could help kids whose first language was Spanish to learn to listen, speak, read, and write in English. This question actually gave birth to the NICHD national “Spanish to English” study carried out in multiple sites across the country.
But during that year it was Goodling and his staff who went to work on the specifics and the need to educate other congressional members not only about the drastic need to address the reading issue, but to emphasize the role of scientific research in solving educational problems. He and his staff devoted substantial time in 1996 reviewing the NICHD reading research. In early 1997, he and his counterparts in the senate held hearings on literacy development and the role of scientific research in developing and implementing effective instructional practices. It came as a surprise to me that in my testimony that year before both House and Senate committees, members asked about research on reading and how it could help guide policy and practice. Their interest in using scientific research to guide practice and policies would later extend to other education programs beyond reading as I was asked to cover the issue in testimony on Title I, Head Start, and IDEA re-authorizations which took place over the next 9 years. And Goodling was the first legislator to formally infuse scientific research in reading into a federal education program. In 1998, He sponsored the Reading Excellence Act, which for the first time required that federal funding be contingent on states and local districts using scientifically based programs.
To further underscore the interest and commitment that congress had in using research to guide federal education policy, Senator Thad Cochran and Representative Anne Northup asked the NICHD in 1998 to convene a National Reading Panel (NRP) to build on the findings of the 1996 NRC panel on preventing reading difficulties in young children. The NRP was tasked to undertake a review of research on reading instruction that would identify the types of programs and principles that were most effective in improving reading proficiency. While the NRC and NRP reports were initiated and published during the Clinton administration, the Bush administration used the findings not only to craft Reading First but to serve as an example of the overarching principle that educational policy and instructional practices should be predicated on research. From this principal evolved the established of the Institute of Educational Sciences, the NRC Report on “Scientific Research in Education”, the Partnership for Reading which served as a resources to disseminate scientific research findings, and the What Works Clearing House. Private groups such as the Council for Excellence in Government, which established the Coalition for Evidence Based Policy, began to contribute to this effort as well.
If you take all of this together, the recent influx of educational science into policy came about through a concerted effort to solve a national reading problem. Using research to guide educational policy and program development has now been extended far beyond reading. A number of actions such as congressional hearings, funding of research reports on science in education, requiring federal funds be contingent on the use of research-based programs and approaches, passing legislation such as the Education Sciences Reform Act of 2002, and building a federal infrastructure which, by its inclusion of the Institute of Educational Sciences and the What Works Clearing House, explicitly sent the message that research-based policies and programs were the rule, not the exception. It is the case that much of the integration of actions and events was strategic and designed to provide a role for scientific research in education. A research to policy and practice culture had to be strengthened through federal legislation and in the scientific infrastructure within the Department of Education.
Time will tell if the gains made in using research to guide education policy will last. History tells us that education is impatient and subject to fads, superstition, anecdotes, and the next magic bullet. To be sure, education is more political than scientific and subject to all the negatives that the political world brings but few of the positives. And many do not understand that by its cannons, evidence is apolitical. There is a tendency to forget that research is not only essential for informing policy but critical for improving policies and programs once in place. But trial and error has become a habit in education and it will take real courage and persistence to overcome that. In a sense, the world of education policy is like a slinky–it can expand to take new steps, but it ultimately recoils back to its original configuration. All this said, I am optimistic.
(Guest Post By Reid Lyon)
I have received many calls from people attending the International Reading Association’s conference in Atlanta informing me that many of the members are celebrating the null findings presented in the Reading First Impact Study Interim Report. But you could have predicted that behavior easily from past behavior. While IRA as an organization has been a supporter of Reading First, many celebrating have wanted the program to fail from day one. But you have to wonder whether the detractors have read the actual interim report or just the press accounts of the evaluation to date? The press coverage overwhelmingly reported the null findings without coverage of the limitations of the study – limitations that should be considered as much as the findings themselves- particularly when drawing any conclusions from the data. There has been little mention of the degree of overlap between Reading First and non-reading First schools and no mention – at least that I can find –of the fact that little time has elapsed since Reading First has been implemented which makes it very difficult to draw conclusions at this time. I am hopeful that these issues will be addressed in detail in the final report given that this information can provide more guidance for improvement. However, in my view and in the whole scheme of things, implementing a program as complex as Reading First will take a bit more time than three years or less.
Indeed, it has been more the rule than the exception that during the first two years of Reading First implementation in districts and schools, teachers were first learning to understand, administer and use the results of assessments to inform instruction.
To jack up the complexity, as they were learning these new concepts, they were also taking part in state reading academies to learn more about the foundations of SBRR (in 5 areas of reading in k-1, in 4 areas of reading in 2-3). To make implementation even more complex, as they were learning and using new assessments and taking part in professional development academies and workshops, they were simultaneously learning how to use new approaches to instruction and how to integrate core program instruction with additional interventions when required to meet individual student needs. This was done at the same time they were learning about center activities, grouping students for instruction and aligning and using supported classroom libraries.
It is important to ask whether any program that has added this amount of new learning to a teacher’s other responsibilities including going to IEP meetings, attending parent conferences, preparing for their instruction in math, social studies and science, serving on school wide committees and a host of other tasks could demonstrate substantial gains after only two years. What is amazing is that despite this unbelievable load, Reading First teachers and their leaders rose to the occasion and have done and are doing a superb job. Also note that the GAO and OMB reports show that they feel that this job is essential and that it is having a major impact.
To be sure, as one of the individuals involved in conceptualizing and drafting the Reading First legislation, it is a no brainer that I am passionate about its potential. That said, the data must speak for the effectiveness of the program. If we wanted to avoid using effectiveness data to monitor and improve the program, we would not have mandated one of the most comprehensive evaluations that have ever been applied to any educational program. The fact that the evaluation plan as written in the legislation was not carried out will require some explanation at some point. More importantly, it is incumbent on both supporters and opponents of Reading First to pay very close attention to the details in the interim evaluation. It is impossible to make informed decisions about improving a program, gutting a program, or reducing the funding for a program on the basis of ambiguous findings. And folks will have to read the details themselves – apparently they will not learn about them from the press.