The Devil’s in the Implementation

May 13, 2008

What went wrong with Reading First?  Don’t blame the evaluation.  Its regression discontinuity design approximates a random assignment experiment — the gold standard of research designs.  It allows us to know with confidence the effect of Reading First on the marginal adopter’s reading achievement.  We can’t assess the effect of Reading First on the first adopters or those who were rated as most in need, but a broadly useful program should have effects beyond those most eager or most desperate.  Reid Lyon is correct in noting that the evaluation did not address everything that we want to know.  And it is always possible that the program needs more time to show results.  But so far we have a null result.

We’re left with two possible explanations.  Either Reading First is conceptually mistaken or it was improperly implemented.  We have good reason to believe that it is the latter.  The science behind Reading First is pretty solid.  A greater emphasis on phonics seems to have a particularly beneficial effect on students from disadvantaged backgrounds. 

Reading First is probably the right idea but as with almost every instructional reform the devil is in the implementation.  The problem is that educators have few incentives to embrace and properly apply new instructional ideas.  It’s not that educators are uninterested in improving instructional approaches.  The problem is that they have often developed approaches from their own experience and training that they think works and are very skeptical of the latest great thing thrown their way.  Any theory of reform that is based on the assumption that educators are eagerly awaiting being informed of what works and will gladly do it once they are told is incredibly naive. 

Even if we could find the right techniques, the difficulty is in getting educators to adopt it and implement it properly.  This is so difficult because teachers don’t experience any meaningful consequences if they properly implement an instructional reform or if they don’t.  And since most teachers have developed routines with which they are comfortable and that they believe are effective, getting them to do something else without any real carrots or sticks is like getting children to eat spinach merely by suggesting it.  You can tell them that it’s really good for them, but they’d rather stick with the familiar mac and cheese.

The evaluation helps confirm that the problem was in implementation.  The differences between the treatment and control groups in time spent on phonics were very small.  And the treatment group was doing far less than the program has planned.  Similar problems have plagues other instructional reforms.  For example, see Mathematica’s evaluation of technology in the classroom, where usage of the technology by the treatment groups was only marginally greater than the control group.  Or see SRI’s evaluation of Following the Leaders, where the treatment group similarly barely used the intervention.  It should come as no surprise that the medicine doesn’t work if people won’t take their pills.

The solution that is usually offered when educators fail to implement an instructional reform is that we need to improve professional development so that they learn better how wonderful the intervention is and why/how they should use it.  Call it education disease — the solution to all problems is more education.  It’s an infinite regress.

Instead the obvious solution is that we have to address the incentives that educators have to adopt and properly implement effective instructional reforms.  Either the direct incentives of accountability with real consequences for teachers (like merit pay or job security) or the indirect incentives of market-based reforms (like school choice) would sharpen educators’ efforts in this regard.

This is why instructional reforms and incentive reforms have to go hand-in-hand.  Educators need to have effective ideas of what to do and they have to have the proper incentives to adopt and implement those effective ideas.  That’s also why pitting instructional reform against incentive reform makes no sense.  We need both.

2 Comments | instructional reform | Tagged: , , | Permalink
Posted by Jay P. Greene

Ask Reid Lyon

May 13, 2008

(Guest Post by Reid Lyon)

“How did scientific research become influential in guiding federal education ‎policy given the field’s historical reliance on ideology, untested ‎assumptions, anecdotes, and superstition to inform both policy and practice?‎”

It has not been an easy journey. In fact it’s like getting a root canal every other week.  What makes it tough is that you are always bumping up against the anti-scientific thinking that has had a misguided influence on the perceived value of research throughout the history of education and increasingly in the past two decades. Many researchers have tried to infuse scientific research into education policy over the years but it never gained political traction. Jeanne Chall gave her career to this cause, but the political will was never there.  Many at the policy level rarely listened to her, much less took her advice.  Chall would tell me frequently that by not basing reading instruction on research we do grave harm to the students education seeks to serve. I repeated her wisdom every time I testified before congressional committees. I also repeated myself time and again that education like other sectors that serve the public, must have reliable information about what works, why it works, and how it works. The alternative was to basically throw mud against a wall and see what sticks – a practice in place for a very long time.  I would argue that scientific research and dissemination of reliable information to the educational community is non-negotiable given that all sectors of a productive society depend on an educated workforce. To be sure, many in the education community sure got medieval on me for holding to this position.

But logic, congressional testimony, research syntheses, or policy papers were not going to change the culture in education which had reinforced an “everything and anything goes” spirit for the past century. Infusing research into policy and practice was going to take strong  support from  a senior  member or members of congress who could argue the need in a compelling way.  Bill Goodling, past chair of the House Education and the Workforce Committee, did just that and in 1996 began to support the concept of “research-based education”.  Goodling was a past educator and was floored when he began to delve into the fact that millions of kids could not read.  His staff learned that the NIH had been studying reading development and reading difficulties since 1965 so they called me in early 1996 to brief the chairman on what we knew about reading from a research standpoint.  At that time, I directed the NICHD Reading Research program at the NIH. During the briefing, he was literally taken aback to learn that NICHD/NIH had studied over 40,000 good and not so good readers, many of them over time, and we had a good idea of what it took to learn to read and what to do about reading difficulties. He could not understand why there was such a massive gap between what research had demonstrated vis-à-vis reading development and instruction and what was actually taught to teachers and  implemented in schools. 

1996 turned out to be a pretty important year in bringing the massive reading failure issue before the public and mobilizing some scientific efforts. It was also an important year for laying the foundation for research-based education policy as it is reflected in federal legislation today.  President Clinton called attention to the tragedy of reading failure in his State of the Union address that year.  His attention to the issue clearly put the problem on congressional radar screens.  In the same year, the Department of Education and the NICHD supported the convening of a National Research Council (NRC) panel to synthesize and summarize research on the prevention of reading difficulties.  Interestingly, at the same time, state leaders were becoming interested in the “research to policy and practice issue”.  Interestingly, in 1996, then Texas governor George Bush asked me and members of several strong research teams in Texas and around the country to brief him on how scientific research in reading could help reduce reading failure in Texas. In one of the meetings he asked a pretty prescient question about how scientific research could help kids whose first language was Spanish to learn to listen, speak, read, and write in English.  This question actually gave birth to the NICHD national “Spanish to English” study carried out in multiple sites across the country.

 But during that year it was Goodling and his staff who went to work on the specifics and the need to educate other congressional members not only about the drastic need to address the reading issue, but to emphasize the role of scientific research in solving educational problems.  He and his staff devoted substantial time in 1996 reviewing the NICHD reading research. In early 1997, he and his counterparts in the senate held hearings on literacy development and the role of scientific research in developing and implementing effective instructional practices. It came as a surprise to me that in my testimony that year before both House and Senate committees, members asked about research on reading and how it could help guide policy and practice.  Their interest in using scientific research to guide  practice and policies would later extend to other education programs beyond reading as I was asked to cover the issue in testimony on Title I, Head Start, and IDEA re-authorizations which took place over the next 9 years.  And Goodling was the first legislator to formally infuse scientific research in reading into a federal education program.  In 1998, He sponsored the Reading Excellence Act, which for the first time required that federal funding be contingent on states and local districts using scientifically based programs.

To further underscore the interest and commitment that congress had in using research to guide federal education policy, Senator Thad Cochran and Representative Anne Northup asked the NICHD in 1998 to convene a National Reading Panel (NRP) to build on the findings of the 1996 NRC panel on preventing reading difficulties in young children.  The NRP was tasked to undertake a review of research on reading instruction that would identify the types of programs and principles that were most effective in improving reading proficiency.  While the NRC and NRP reports were initiated and published during the Clinton administration, the Bush administration used the findings not only to craft Reading First but to serve as an example of the overarching principle that educational policy and instructional practices should be predicated on research.  From this principal evolved the established of the Institute of Educational Sciences, the NRC Report on “Scientific Research in Education”, the Partnership for Reading which served as a resources to disseminate scientific research findings, and the What Works Clearing House.  Private groups such as the Council for Excellence in Government, which established the Coalition for Evidence Based Policy,  began to contribute to this effort as well.

If you take all of this together, the recent influx of educational science into policy came about through a concerted effort to solve a national reading problem. Using research to guide educational policy and program development has now been extended far beyond reading.   A number of actions such as congressional hearings, funding of research reports on science in education, requiring federal funds be contingent on the use of research-based programs and approaches, passing legislation such as the Education Sciences Reform Act of 2002, and building a federal infrastructure which, by its inclusion of the Institute of Educational Sciences and the What Works Clearing House, explicitly sent the message that research-based policies and programs were the rule, not the exception.  It is the case that much of the integration of actions and events was strategic and designed to provide a role for scientific research in education.  A research to policy and practice culture had to be strengthened through federal legislation and in the scientific infrastructure within the Department of Education. 

Time will tell if the gains made in using research to guide education policy will last.  History tells us that education is impatient and subject to fads, superstition, anecdotes, and the next magic bullet.  To be sure, education is more political than scientific and subject to all the negatives that the political world brings but few of the positives. And many do not understand that by its cannons, evidence is apolitical.  There is a tendency to forget that research is not only essential for informing policy but critical for improving policies and programs once in place. But trial and error has become a habit in education and it will take real courage and persistence to overcome that.  In a sense, the world of education policy is  like a slinky–it can expand to take new steps, but it ultimately recoils back to its original configuration.   All this said, I am optimistic.

12 Comments | instructional reform, methodology, politics | Tagged: , , | Permalink
Posted by Jay P. Greene

Reading First — There’s More to the Story

May 8, 2008

(Guest Post By Reid Lyon)

I have received many calls from people attending the International Reading Association’s conference in Atlanta informing me that many of the members are celebrating the null findings presented in the Reading First Impact Study Interim Report.  But you could have predicted that behavior easily from past behavior.  While IRA as an organization has been a supporter of Reading First, many celebrating have wanted the program to fail from day one.  But you have to wonder whether the detractors have read the actual interim report or just the press accounts of the evaluation to date?   The press coverage overwhelmingly reported the null findings without coverage of the limitations of the study – limitations that should be considered as much as the findings themselves- particularly when drawing any conclusions from the data.  There has been little mention of the degree of overlap between Reading First and non-reading First schools and no mention – at least that I can find –of the fact that little time has elapsed since Reading First has been implemented which makes it very difficult to draw conclusions at this time.   I am hopeful that these issues will be addressed in detail in the final report given that this information can provide more guidance for improvement.   However, in my view and in the whole scheme of things, implementing a program as complex as Reading First will take a bit more time than three years or less.

Indeed, it has been more the rule than the exception that during the first two years of Reading First implementation in districts and schools, teachers were first learning to understand, administer and use the results of assessments to inform instruction.

To jack up the complexity, as they were learning these new concepts, they were also taking part in state reading academies to learn more about  the foundations of SBRR (in 5 areas of reading in k-1, in 4 areas  of reading in 2-3). To make implementation even more complex, as they were learning and using new assessments and taking part in professional development academies and workshops, they were simultaneously learning how to use new approaches to instruction and how to integrate core program instruction with additional interventions when required to meet individual student needs. This was done at the same time they were learning about center activities, grouping students for instruction and aligning and using supported classroom libraries.

It is important to ask whether any program that has added this amount of new learning to a teacher’s other responsibilities including going to IEP meetings, attending parent conferences, preparing for their instruction in math, social studies and science, serving on school wide committees and a host of other tasks could demonstrate substantial gains after only two years. What is amazing is that despite this unbelievable load, Reading First teachers and their leaders rose to the occasion and have done and are doing a superb job. Also note that the GAO and OMB reports show that they feel that this job is essential and that it is having a major impact.

To be sure, as one of the individuals involved in conceptualizing and drafting the Reading First legislation, it is a no brainer that I am passionate about its potential.  That said, the data must speak for the effectiveness of the program.  If we wanted to avoid using effectiveness data to monitor and improve the program, we would not have mandated one of the most comprehensive evaluations that have ever been applied to any educational program.   The fact that the evaluation plan as written in the legislation was not carried out will require some explanation at some point.  More importantly, it is incumbent on both supporters and opponents of Reading First to pay very close attention to the details in the interim evaluation.  It is impossible to make informed decisions about improving a program, gutting a program, or reducing the funding for a program on the basis of ambiguous findings.  And folks will have to read the details themselves – apparently they will not learn about them from the press.

3 Comments | instructional reform | Tagged: , , | Permalink
Posted by Jay P. Greene

Reading First Disappoints

May 1, 2008

A new study released today by the U.S. Department of Education finds that Reading First, the phonics based federally-funded reading program, failed to yield improvements in reading comprehension.  The study doesn’t demonstrate that it is ineffective to emphasize phonics; only that it was ineffective to adopt this particular program.  That could be because there are problems with the design or implementation of Reading First.  Or it could be because the control group was also receiving phonics instruction without the particular elements of Reading First.  And it is possible that phonics is less effective than some people thought.

Whatever the explanation, this well-designed study undermines confidence that instructional reform, like Reading First, can alone transform the educational system

16 Comments | instructional reform, research reports | Tagged: , , , | Permalink
Posted by Jay P. Greene

Manipulatives Make Math Mushy

April 25, 2008

An interesting item in this morning’s New York Times — Someone has finally done an experimental study on the math instructional technique that emphasizes the use of blocks, balls, and other concrete “manipulatives” to teach math.  Researchers at Ohio State University created an experiment in which they randomly assigned subjects to be taught a new math concept either by focusing on the abstract math rule, focusing on the use of manipulatives, or combining both techniques.  They then tested how well subjects had learned the math concept by having them apply it to a new situation.  It turns out that students taught with manipulatives did the worst, the ones taught abstractly did the best, and the combined approach performed in the middle.  It appears that those taught math with more concrete examples had a harder time transferring that math concept to a different concrete example.  Kids taught math with tennis balls have a harder time applying the principle to railroad cars.

4 Comments | instructional reform, research reports | Tagged: , | Permalink
Posted by Jay P. Greene

Next Entries »