Reading First Final Report

November 19, 2008

The final report of the U.S. Department of Education commissioned evaluation of Reading First was released yesterday.  According to Russ Whitehurst, the USDOE’s director of the Institute for Education Sciences: “I don’t think anyone should be celebrating that the federal government has spent $6 billion on a reading program that has had no impact on reading comprehension.”

For more perspective on Reading First, people may want to check out Reid Lyon’s posts on the program on this blog, including this one.  I also have commentary here.


The Infinte Regress

October 29, 2008

There is no problem to which more education is not the proposed solution.  Teachers aren’t as effective as they should be?  Increase professional development.  Professional development isn’t as effective as it should be?  Increase training for providers of professional development.  Wash.  Rinse.  Repeat.

So, when Mathematica found that intensive mentoring for first year teachers had no effect on those teachers’ practices or their students’ academic achievement, what did folks have to say?  Improve the training of the mentors

Similarly, when Mathematica evaluated a broad range of education technology in schools they found: “Test scores were not significantly higher in classrooms using selected reading and mathematics software products. Test scores in treatment classrooms that were randomly assigned to use products did not differ from test scores in control classrooms by statistically significant margins.”  But, critics of the study said that it “didn’t take into account the critical factors of proper implementation and curriculum integration, professional development for teachers, planning, or infrastructure issues, among others. ”  That is, the results would be better if only we provided more education to teachers and administrators to implement the technology appropriately.  Wash. Rinse. Repeat.

And again when the Department of Education’s evaluation of Reading First showed no advantage for students’ reading achievement, others responded that the schools studied had not properly implemented the program or trained their teachers.

The problem with offering more education as the solution to each failure is that it assumes that the only thing educators are lacking is knowledge of the right thing to do.  If only we bother to tell them, educators are hungry to learn the right thing and implement it well.  But as I’ve argued in the past, educators are also lacking the motivation to learn these techniques and implement them well.

All of these interventions — mentoring, technology, and increased reliance on phonics — may very well be desirable.  But unless we address the incentives that educators have to identify effective practices, learn them, and use them well, no amount of additional education will solve the problem.


Learn to Swim by Drowning

July 10, 2008

(Guest post by Greg Forster)

So far I’ve kept largely out of the hubbub over Reading First, but I can’t resist a comment on Stephen Krashen’s “opposing view” editorial in yesterday’s USA Today.

On the merits of Reading First I offer no opinion, but Krashen’s proposal for what we should be doing instead of Reading First is a good illustration of how little the program’s opponents are offering by way of promising alternatives.

Krashen argues that we should increase literacy by spending more money on libraries. Apparently the mere physical presence of books will help people learn how to read – by osmosis, presumably.

Actually, Krashen’s real argument is that we don’t have a problem with literacy anyway. He asserts that 99 percent of U.S. adults can read and write “on a basic level.” Thus, we should be focused on increasing people’s ability to read at a higher level – in which respect he asserts that the main obstacle is a lack of access to reading material among low-income populations.

His source for the 99 percent literacy datum is the CIA World Factbook. Insert your own joke about the CIA’s “slam dunk” intelligence on Saddam’s WMD program here.

In fact, the CIA includes all persons 15 years and older in this statistic, so we’re not just talking about “adults.” The CIA is actually claiming that 99 percent of U.S. adults and teenagers can read and write.

Clearly the CIA is defining “literacy” at such a low level as to be meaningless for evaluating the need for programs like Reading First. If the CIA considers 99 percent of U.S. adults and teenagers to be literate, then it must be counting the ability to read a stop sign as literacy. Reading First is intended to address literacy problems on a slightly more serious level.

As for the idea of spending more money on libraries, if the unspecified “studies” that Krashen asserts show literacy benefits from libraries involve scientifically valid analysis of systematically collected empirical data, then by all means let’s spare a little more money for the libraries. (In Krashen’s defense on that last point, USA Today doesn’t really offer a lot of space for specifics on what studies you’re referring to and what methods they used.)

But the alleged need for more libraries really ought to be considered separately from fights over pedagogy. The relevant question for evaluating the merits of Reading First is how we ought to be teaching reading in our schools. Unless Krashen wants to quit teaching reading in schools and just lock the kids in the library until they figure out how to read, his argument for more libraries really doesn’t speak to the question at hand. He might as well argue that since Reading First allegedly doesn’t work, we should be spending the money on hospitals instead.

All of this, of course, is separate from the question of whether spending more money on libraries really would improve literacy. In response to Krashen’s editorial, I recieved an e-mail that was circulated by Martin Kozloff of the education school at the University of North Carolina-Wilmington, who gave me permission to post it here:

This [proposal to spend more on libraries] has been Krashen’s refrain for the past 20+ years. It’s hard to argue with investment in libraries, but it feels like a bait and switch argument rather than addressing instructional needs of students.

Yes, we CAN eliminate drowning simply by building more swimming pools and then immersing the kids in water-rich environments.

“Here, ya go, Billy.” [SPLASH]

“HAAALLLLP!”

“It’s alright, Billy, you are an emergent swimmer.”

Kozloff takes quotes from critics of phonics-based instruction and substitutes swimming for reading:

“Children must develop [swimming] strategies by and for themselves.”

“Saying that we are determined to teach every child to [swim] does not mean that we will teach every child to [swim]…The best we can do … is … to ensure that, if not every child [survives a rip tide], there is a minimum of guilt and anguish on the part of teachers, students, and parents.”

“We might offer students some [floating] hints at an appropriate moment when they are [drowning] and aren’t sure how to [stay afloat].”

“[Swimming] learning proceeds naturally if the environment supports young children’s experimentation with [rip currents].”

“In my view, [swimming] is not a matter of [stroking with your arms and kicking with your legs] but of bringing meaning to [drowning].”

“Early in our miscue research, we concluded.That [the middle of an ocean] is easier to [swim in] than a [raging river], a [raging river] easier to [swim in] than a [lake] , a [lake] easier than a [pool], a [pool] easier than a [bath tub], and a [bath tub] easier than a [kitchen sink]. Our research continues to support this conclusion and we believe it to be true.”

“The worst [swimmers] are those who try to [paddle and kick] according to the rules of [physics and common sense].””In my view, [swimming] is not a matter of [stroking with your arms and kicking with your legs] but of bringing meaning to [drowning].”

“Early in our miscue research, we concluded.That [the middle of an ocean] is easier to [swim in] than a [raging river], a [raging river] easier to [swim in] than a [lake] , a [lake] easier than a [pool], a [pool] easier than a [bath tub], and a [bath tub] easier than a [kitchen sink]. Our research continues to support this conclusion and we believe it to be true.”

“The worst [swimmers] are those who try to [paddle and kick] according to the rules of [physics and common sense].” 

Kozloff is, of course, a harsh critic of whole language instruction. I have no desire to step into the phonics/whole language debate as such.

But Kozloff clearly has a point when he observes that offering libraries as an alternative to Reading First is like offering swimming pools as an alternative to a program of swimming lessons. Even if the lessons in the Swimming First program aren’t effective, it’s simply a distraction to respond by talking about the need for more swimming pools.

And quite a lot of the noise about Reading First has this quality about it – by which I mean what Kozloff calls a “bait and switch” quality. No one in the national spotlight seems to be championing whole language the way they were, say, ten years ago. If the critics think whole language is the way we should go, let them say so. If not, what are their alternative models for good pedagogy?


Ask Reid Lyon – Reading First Implementation

May 19, 2008

(Guest post by Reid Lyon)

“The implementation of Reading First has been a hot topic, especially since the release of the recent impact study. Has the implementation of the program been successful?  What were the most important issues that arose regarding the implementation of Reading First? What lessons have been learned?”

 

With respect to whether the implementation of the program has been successful, the short answer is it depends.  Amber Winkler, the research director for the Fordham Foundation, recently reported in the Gadfly that Reading First is “perhaps the best-implemented education program in federal history.”  This may be true in some states, but not in others.  Dr. Winkler highlights Georgia, Oregon, and California as states that got the implementation process right.

 

On the other hand, Texas and several other states made successful implementation unbearably difficult.  In Texas, for example, a process was initially in place to award districts Reading First funding if their proposals were rated above a particular score by the grant review group.  Proposals from districts scoring below a threshold value were to be resubmitted following technical assistance to equip the district with in depth understanding of Scientifically Based Reading Research (SBRR) and to incorporate those concepts in their resubmission along with assessment and accountability requirements.  For some reason, a decision was made to award all districts Reading First funding irrespective of the quality of their applications.  Moreover, eligible districts received Reading First funding before any technical assistance was in place and before any baseline reading assessments could be administered for program evaluation purposes.  Not good.

 

But there is a great deal more involved in the implementation of a complex program beyond ensuring the quality of grant applications, and few states and districts had their hands around all of the essential conditions that must be in place to embed and bring to scale an initiative as intricate as Reading First.  At first blush, it would seem that the existence of a converging body of evidence relevant to reading development, reading difficulties, and reading instruction would have facilitated implementation fidelity.  But a substantial research base, bipartisan political support, and hefty funding only go so far.  The devil is in the details, and strategies for implementation at the federal level and in many states either were not appraised of the details or simply felt, as in Texas, that the money had to flow immediately no matter what, reflecting education’s love affair with entitlement programs.

 

One detail that gets right in your face immediately when you are implementing a program as complex as Reading First is that you have to manage coordinated systems change at federal, state, district, school and classroom levels.  Complexity theorists like W.L. Miller and his team use the metaphor of a jazz band when discussing how individuals within an effective system perform their own tasks in concert with others in achieving a desired goal.  Each contributor is responsible for certain tasks, but always listening to the other members of the orchestra to determine how their own actions contribute to the whole. This is tough to do when the band members come from different generations and different musical perspectives.  Getting into the groove takes a good deal of practice and a willingness to expand one’s thinking.  These are not features that characterize public education.   Basically, any evaluation of a program like Reading First must drill down into how this coordination played out and how long it took.  

 

Another common-sense detail that was not planned for in the implementation schedule was the fact that many hard-working folks in schools and classrooms who have gotten used to doing things in a certain way were being asked to change their routines and to try something new. In some cases this meant a decision was made to stop certain programs and replace them with others that many were unfamiliar with. When this occurs, teachers and leaders must have confidence that what is being implemented provides advantages to students over existing practices – and, in many instances, the case for this was not made prior to implementation.  While this may seem a bit fluffy, it is important to understand that public-school educators are under a constant barrage of new magic bullets, fads, and aggressive textbook company representatives all selling materials “based on SBRR.”  Implementation experts will tell you that without teacher and leader buy-in, any program, no matter how effective, will not realize its potential.  Fortunately, the majority of those leading and teaching in Reading First districts and schools saw the clear advantages offered students by the program as they observed poor readers become good readers.  But this took a while. 

 

In my interactions with many Reading First programs over the past six years, I did notice some common conditions that were in place when implementation fidelity was strong.  In addition to the details noted above, strong implementers embraced data and accountability for results.  States like Alabama integrated robust professional development with continuous coaching and feedback for both teachers and leaders.  Instructional programs were selected not only on the basis of their alignment with SBRR but because they were practical, useful, and beneficial to students.  Teachers were treated as self-determined professionals and responded by taking ownership and responsibility for their parts in the jazz band.  And building-level leadership ensured that teachers and coaches had the necessary time to plan, review student data, and collaborate in differentiating instruction for individual students based on their performance data.

 

The lessons learned are many, but I can think of three big ones.  Let’s start with the way Congress and the feds typically expect complex programs to be in place, in full operation, as soon legislation is passed.  Reading First embodied so many new concepts and requirements that, in my view, the first year should have been spent in providing technical assistance and professional development to states and districts even prior to the submission of Reading First grants.  I can’t tell you the number of times I saw the thousand-yard stare following my mention of SBRR, progress monitoring, data-driven instruction, or comprehensive reading programs.  We are talking significant mismatch between the requirements of the legislation and the background knowledge of many grant applicants; not to mention that the grants were competitive – a novel concept in education formula funding.

 

And if you were on the ground during the first two years of the program, this is what you would typically observe:  Teachers were first learning to understand, administer and use the results of assessments to inform instruction.  As they were learning these new concepts, they were also taking part in state reading academies to learn more about  the foundation of SBRR (in 5 areas of reading in k-1, in 4 areas  of reading in 2-3). In addition, as they were learning and using new assessments and taking part in professional development academies and workshops, they were simultaneously learning how to use a new approach to instruction and how to integrate core program instruction with additional interventions when required to meet individual student needs. This was done at the same time they were learning about center activities, grouping students for instruction and aligning and using supported classroom libraries.

 

It is important to ask whether any program that has added this amount of new learning to a teacher’s other responsibilities – including going to IEP meetings, attending parent conferences, preparing for their instruction in math, social studies and science, serving on school-wide committees and a host of other tasks – could demonstrate substantial gains after three years.  Give me a break.  What is amazing is that despite this unbelievable load, Reading First coordinators, teachers and their leaders rose to the occasion and have done and are doing a superb job.

 

Lesson Number One: Take a year to develop the infrastructure essential for program implementation. 

 

Lesson Number Two: During this first year,  make sure that all involved at every level understand the essential conditions that have to be in place to coordinate and implement a massive and unique program and to anticipate the need to customize some of its features based on individual district and school characteristics.

 

Lesson Number Three – and this is for the Department of Education: The next time Congress gives you $25 million dollars a year for six years to carry out an ongoing evaluation of a program, for God’s sake design and implement the evaluation commensurate with the initiation of the program.  This was no time to carry out a delayed and abbreviated evaluation when the complexity and uniqueness of a program demanded comprehensive, continuous and systematic feedback to ensure improvements in implementation where needed.


Hans Brix? Oh no!

May 14, 2008

(Guest Post by Matthew Ladner)

In Team America: World Police Hans Blix shows up at the palace of Kim Jong Il. Blix demands to inspect North Korean nuclear facilities, “or else.” Kim Jong Il asks or else what, and Blix threatens to send “a very angry letter” from the United Nations. Kim proceeds to open a trap door, dropping Blix into a shark tank.

I would include a youtube link, but this is a family blog.

So just to sum up what I have gathered on Reading First: we have decades worth of high quality evidence showing that public schools are using terribly ineffective reading methods. When the Bush administration finally tries to do something about it, with serious money involved, lobbyists water down the bill language and the program administrator is subjected to a witch hunt. Essentially the schools take the billions and barely implement the program. When the program is evaluated, it “doesn’t work.”

The next time one of my fellow reformers suggests that they can fix things once they get to be the ones with their firm grip on the ship wheel, I’ll humbly suggest that they have the phrase “READING FIRST” tattooed to their forehead to serve as a constant reminder of how education policy actually works. Meaningful education reform can be done, but it works best when there is pressure from both the top down and the bottom up.


The Devil’s in the Implementation

May 13, 2008

What went wrong with Reading First?  Don’t blame the evaluation.  Its regression discontinuity design approximates a random assignment experiment — the gold standard of research designs.  It allows us to know with confidence the effect of Reading First on the marginal adopter’s reading achievement.  We can’t assess the effect of Reading First on the first adopters or those who were rated as most in need, but a broadly useful program should have effects beyond those most eager or most desperate.  Reid Lyon is correct in noting that the evaluation did not address everything that we want to know.  And it is always possible that the program needs more time to show results.  But so far we have a null result.

We’re left with two possible explanations.  Either Reading First is conceptually mistaken or it was improperly implemented.  We have good reason to believe that it is the latter.  The science behind Reading First is pretty solid.  A greater emphasis on phonics seems to have a particularly beneficial effect on students from disadvantaged backgrounds. 

Reading First is probably the right idea but as with almost every instructional reform the devil is in the implementation.  The problem is that educators have few incentives to embrace and properly apply new instructional ideas.  It’s not that educators are uninterested in improving instructional approaches.  The problem is that they have often developed approaches from their own experience and training that they think works and are very skeptical of the latest great thing thrown their way.  Any theory of reform that is based on the assumption that educators are eagerly awaiting being informed of what works and will gladly do it once they are told is incredibly naive. 

Even if we could find the right techniques, the difficulty is in getting educators to adopt it and implement it properly.  This is so difficult because teachers don’t experience any meaningful consequences if they properly implement an instructional reform or if they don’t.  And since most teachers have developed routines with which they are comfortable and that they believe are effective, getting them to do something else without any real carrots or sticks is like getting children to eat spinach merely by suggesting it.  You can tell them that it’s really good for them, but they’d rather stick with the familiar mac and cheese.

The evaluation helps confirm that the problem was in implementation.  The differences between the treatment and control groups in time spent on phonics were very small.  And the treatment group was doing far less than the program has planned.  Similar problems have plagues other instructional reforms.  For example, see Mathematica’s evaluation of technology in the classroom, where usage of the technology by the treatment groups was only marginally greater than the control group.  Or see SRI’s evaluation of Following the Leaders, where the treatment group similarly barely used the intervention.  It should come as no surprise that the medicine doesn’t work if people won’t take their pills.

The solution that is usually offered when educators fail to implement an instructional reform is that we need to improve professional development so that they learn better how wonderful the intervention is and why/how they should use it.  Call it education disease — the solution to all problems is more education.  It’s an infinite regress.

Instead the obvious solution is that we have to address the incentives that educators have to adopt and properly implement effective instructional reforms.  Either the direct incentives of accountability with real consequences for teachers (like merit pay or job security) or the indirect incentives of market-based reforms (like school choice) would sharpen educators’ efforts in this regard.

This is why instructional reforms and incentive reforms have to go hand-in-hand.  Educators need to have effective ideas of what to do and they have to have the proper incentives to adopt and implement those effective ideas.  That’s also why pitting instructional reform against incentive reform makes no sense.  We need both.


Ask Reid Lyon

May 13, 2008

(Guest Post by Reid Lyon)

“How did scientific research become influential in guiding federal education ‎policy given the field’s historical reliance on ideology, untested ‎assumptions, anecdotes, and superstition to inform both policy and practice?‎”

It has not been an easy journey. In fact it’s like getting a root canal every other week.  What makes it tough is that you are always bumping up against the anti-scientific thinking that has had a misguided influence on the perceived value of research throughout the history of education and increasingly in the past two decades. Many researchers have tried to infuse scientific research into education policy over the years but it never gained political traction. Jeanne Chall gave her career to this cause, but the political will was never there.  Many at the policy level rarely listened to her, much less took her advice.  Chall would tell me frequently that by not basing reading instruction on research we do grave harm to the students education seeks to serve. I repeated her wisdom every time I testified before congressional committees. I also repeated myself time and again that education like other sectors that serve the public, must have reliable information about what works, why it works, and how it works. The alternative was to basically throw mud against a wall and see what sticks – a practice in place for a very long time.  I would argue that scientific research and dissemination of reliable information to the educational community is non-negotiable given that all sectors of a productive society depend on an educated workforce. To be sure, many in the education community sure got medieval on me for holding to this position.

But logic, congressional testimony, research syntheses, or policy papers were not going to change the culture in education which had reinforced an “everything and anything goes” spirit for the past century. Infusing research into policy and practice was going to take strong  support from  a senior  member or members of congress who could argue the need in a compelling way.  Bill Goodling, past chair of the House Education and the Workforce Committee, did just that and in 1996 began to support the concept of “research-based education”.  Goodling was a past educator and was floored when he began to delve into the fact that millions of kids could not read.  His staff learned that the NIH had been studying reading development and reading difficulties since 1965 so they called me in early 1996 to brief the chairman on what we knew about reading from a research standpoint.  At that time, I directed the NICHD Reading Research program at the NIH. During the briefing, he was literally taken aback to learn that NICHD/NIH had studied over 40,000 good and not so good readers, many of them over time, and we had a good idea of what it took to learn to read and what to do about reading difficulties. He could not understand why there was such a massive gap between what research had demonstrated vis-à-vis reading development and instruction and what was actually taught to teachers and  implemented in schools. 

1996 turned out to be a pretty important year in bringing the massive reading failure issue before the public and mobilizing some scientific efforts. It was also an important year for laying the foundation for research-based education policy as it is reflected in federal legislation today.  President Clinton called attention to the tragedy of reading failure in his State of the Union address that year.  His attention to the issue clearly put the problem on congressional radar screens.  In the same year, the Department of Education and the NICHD supported the convening of a National Research Council (NRC) panel to synthesize and summarize research on the prevention of reading difficulties.  Interestingly, at the same time, state leaders were becoming interested in the “research to policy and practice issue”.  Interestingly, in 1996, then Texas governor George Bush asked me and members of several strong research teams in Texas and around the country to brief him on how scientific research in reading could help reduce reading failure in Texas. In one of the meetings he asked a pretty prescient question about how scientific research could help kids whose first language was Spanish to learn to listen, speak, read, and write in English.  This question actually gave birth to the NICHD national “Spanish to English” study carried out in multiple sites across the country.

 But during that year it was Goodling and his staff who went to work on the specifics and the need to educate other congressional members not only about the drastic need to address the reading issue, but to emphasize the role of scientific research in solving educational problems.  He and his staff devoted substantial time in 1996 reviewing the NICHD reading research. In early 1997, he and his counterparts in the senate held hearings on literacy development and the role of scientific research in developing and implementing effective instructional practices. It came as a surprise to me that in my testimony that year before both House and Senate committees, members asked about research on reading and how it could help guide policy and practice.  Their interest in using scientific research to guide  practice and policies would later extend to other education programs beyond reading as I was asked to cover the issue in testimony on Title I, Head Start, and IDEA re-authorizations which took place over the next 9 years.  And Goodling was the first legislator to formally infuse scientific research in reading into a federal education program.  In 1998, He sponsored the Reading Excellence Act, which for the first time required that federal funding be contingent on states and local districts using scientifically based programs.

To further underscore the interest and commitment that congress had in using research to guide federal education policy, Senator Thad Cochran and Representative Anne Northup asked the NICHD in 1998 to convene a National Reading Panel (NRP) to build on the findings of the 1996 NRC panel on preventing reading difficulties in young children.  The NRP was tasked to undertake a review of research on reading instruction that would identify the types of programs and principles that were most effective in improving reading proficiency.  While the NRC and NRP reports were initiated and published during the Clinton administration, the Bush administration used the findings not only to craft Reading First but to serve as an example of the overarching principle that educational policy and instructional practices should be predicated on research.  From this principal evolved the established of the Institute of Educational Sciences, the NRC Report on “Scientific Research in Education”, the Partnership for Reading which served as a resources to disseminate scientific research findings, and the What Works Clearing House.  Private groups such as the Council for Excellence in Government, which established the Coalition for Evidence Based Policy,  began to contribute to this effort as well.

If you take all of this together, the recent influx of educational science into policy came about through a concerted effort to solve a national reading problem. Using research to guide educational policy and program development has now been extended far beyond reading.   A number of actions such as congressional hearings, funding of research reports on science in education, requiring federal funds be contingent on the use of research-based programs and approaches, passing legislation such as the Education Sciences Reform Act of 2002, and building a federal infrastructure which, by its inclusion of the Institute of Educational Sciences and the What Works Clearing House, explicitly sent the message that research-based policies and programs were the rule, not the exception.  It is the case that much of the integration of actions and events was strategic and designed to provide a role for scientific research in education.  A research to policy and practice culture had to be strengthened through federal legislation and in the scientific infrastructure within the Department of Education. 

Time will tell if the gains made in using research to guide education policy will last.  History tells us that education is impatient and subject to fads, superstition, anecdotes, and the next magic bullet.  To be sure, education is more political than scientific and subject to all the negatives that the political world brings but few of the positives. And many do not understand that by its cannons, evidence is apolitical.  There is a tendency to forget that research is not only essential for informing policy but critical for improving policies and programs once in place. But trial and error has become a habit in education and it will take real courage and persistence to overcome that.  In a sense, the world of education policy is  like a slinky–it can expand to take new steps, but it ultimately recoils back to its original configuration.   All this said, I am optimistic.


Follow

Get every new post delivered to your Inbox.

Join 2,567 other followers