A new study released today by the U.S. Department of Education finds that Reading First, the phonics based federally-funded reading program, failed to yield improvements in reading comprehension. The study doesn’t demonstrate that it is ineffective to emphasize phonics; only that it was ineffective to adopt this particular program. That could be because there are problems with the design or implementation of Reading First. Or it could be because the control group was also receiving phonics instruction without the particular elements of Reading First. And it is possible that phonics is less effective than some people thought.
Whatever the explanation, this well-designed study undermines confidence that instructional reform, like Reading First, can alone transform the educational system
Jay P. Greene's Blog 
I was struck by this from page 44:
So, if you add an extra three to five minutes of phonics/comprehension, and subtract four to eleven minutes of vocabulary and fluency, there won’t be a huge improvement? That’s what this study is reporting?
The fact that the difference in time devoted to phonics caused by adopting Reading First is small suggests either that 1) Reading First was designed or implemented such that there was an insufficient amount of time devoted to phonics, or 2) an emphasis on phonics is already widespread (remember that the study was based on 17 districts and 1 entire state) so that adopting a program explicitly focused on phonics produces little change in time devoted. Either way we shouldn’t expect (and didn’t get) much improvement by adopting Reading First.
But remember this is the program that many touted as the thing that would cure what ails us.
True. But I guess what I’m thinking is this: If Reading First were shown to increase the time spent on various reading tasks by really substantial amounts, and that still made no difference, then it would seem a waste of time to try to get anyone to focus on those reading tasks. But if Reading First hardly makes any difference in what teachers actually do, then I don’t know what that means. Does that mean it’s superfluous to try to change what teachers actually do (whether because they’re all intransigent or because they were doing the right stuff anyway)? Or does it mean that we should try harder to implement instructional reform?
It might nean either of those things. To find out, you’d have to try another experiment, say with a reform that’s more aggressive in seeking to change teacher behavior.
Mike Petrilli sees problems with the study over at Flypaper (http://www.edexcellence.net/flypaper/index.php/2008/05/on-reading-first-read-the-report-first/ )
I don’t find his argument fully persuasive and left a comment there.
[…] Greene says it’s a well-designed study. The lack of effect may reflect weak implementation or the fact that some control-group schools […]
The study link does not work for me.
If you are having trouble with the link to the study, try this:
http://ies.ed.gov/ncee/pubs/20074007/execsumm_background.asp
I would not as far as to say that the Reading First Impact study was wll designed given its congressional charge and the language detailing its need to be comprehensive (see Section 1205) of the Reading First legislation. That fact is that it was delayed in its design and impelemtation for inexplicable reasons. The delay in designing and implementing the impact evaluation intended by Congress is not a trivial matter for several reasons.
First, Bob Sweet and I in drafting the legislation felt that a comprehensive and rigorous evaluation of the Reading First program was essential. In fact, we felt that it was imperative that the evaluation must have the necessary scope, transparency, and funding to provide the fine grained details essential for program improvement and to also establish a model for future federal impact studies. For this reason, we crafted 10 primary analyses and measurements that an independent evaluator was required to carry out (again see Section 1205 of the law). Specifically the law required:
1) An analysis of the relationship between each of the essential components of reading instruction and overall reading proficiency.
(2) An analysis of whether assessment tools used by State educational agencies and local educational agencies measure the essential components of reading.
(3) An analysis of how State reading standards correlate with the essential components of reading instruction.
(4) An analysis of whether the receipt of a targeted assistance grant under section 1204 results in an increase in the number of children who read proficiently.
(5) A measurement of the extent to which specific instructional materials improve reading proficiency.
(6) A measurement of the extent to which specific screening, diagnostic, and classroom-based instructional reading assessments assist teachers in identifying specific reading deficiencies.
(7) A measurement of the extent to which professional development programs implemented by State educational agencies using funds received under this subpart improve reading instruction.
(8) A measurement of how well students preparing to enter the teaching profession are prepared to teach the essential components of reading instruction.
(9) An analysis of changes in students’ interest in reading and time spent reading outside of school
(10) Any other analysis or measurement pertinent to this subpart that is determined to be appropriate by the Secretary.
Because these are very complex analyses requiring research designs and methods (including sampling strategies) appropriate addressing each evaluation target, the law provided $25 million dollars PER YEAR over a six year period (total = $150 million) to ensure that the evaluation research tasks could be accomplished. The amount of funds set aside for the external evaluation was arrived at following a survey of evaluation researchers who were asked to identify the cost of the most rigorous evaluation possible. Twenty-five million dollars per year was the arrived at figure.
Second, the delay led to an inability to acquire the data essential to addressing each of the required analyses and measurements required in the legislation. As it stands now, it is difficult, if not impossible to identify which reading components were most highly related to comprehension outcomes, which instructional programs and materials had the most significant impact on every reading component including comprehension, whether students engaged in more outside reading as a function of the reading instruction they were receiving, whether professional development activities and content were significantly related to reading outcomes in both Reading First and non-Reading First schools, and whether critical implementation factors accounted for variation in reading outcomes. Had the evaluation been ready to be implemented in the early stages of Reading First the amount of critical information that we would have learned would have increased dramatically.
Third, unless the final Impact Study report addresses the required analyses and measurements articulated in the legislation and listed above, it is not the study that the Congress intended. A significant amount of information will have been lost. More importantly, it will not be possible to answer a fundamental question: For which students are which instructional materials (programs) most beneficial under well defined conditions (professional development, implementation fidelity, and so on).
It is unclear why the evaluation was delayed or not carried out to meet the requirements in the legislation. The legislation, with the evaluation requirements included in the law, was passed prior to the time the IES director came on board. It could be that the critical need to design and implement the evaluation rapidly was lost between the cracks somewhere, but there is documented correspondence from the Education and Workforce Committee inquiring about delays. Whatever the reason(s), the results from the interim evaluation are very difficult to interpret, particularly if one is trying to inform improvements in both the policy and the program. I have addressed a number of concerns about the interim evaluation report which can be found at:
http://www.EdNews.org
Interview with Reid Lyon: Reading First is the largest concerted reading intervention program in the history of the civilized world
Thanks for your comment, Dr. Lyon. By saying that the study was “well-designed” I was referring to the rigor of a regression discontinuity design for assessing the overall impact of the program as implemented on the reading comprehension measures employed. Regression discontinuity approximates a random assignment design, which is the gold standard of evaluation research.
I recognized that the program may not have been implemented properly. And you raise several other points that we should keep in mind as we interpret the results.
I agree that a regression-discontinuity design was appropriate for the four questions posed in the evaluation, but the sampling issues and potential for contamination across Reading First and non-Reading First schools is pretty hefty. Likewise, the lack of specific tests of relationships between components of reading and comprehension outcomes and the toal absence of tests to addresses effectiveness for specific programs and materials is troublesome. I am still at a loss to understand a why a comprehensive and genuinely informative evaluation was not accomplished, particularly when it was the clear congressional intent backed up by the funding to get the job done. Sometimes I do wonder whether the Department of Education has an autoimmune disease given its persistent nature to attack itself from within.
I also find that Stuart Buck has raised very important points. The interim report presented data that the actual time spent in daily instruction in the five reading domains in Reading First classrooms was in the 59 minute range and about 50 minutes in the non-Reading First schools. While this is statistically significant, I have my doubts that 9 minutes of actual daily instruction will produce a great deal of difference.
Moreover, it is surprising that the instructional time in these schools is far below what state level Reading First implementations require, which hovers around the 90 minutes of core reading instruction and an additional period (usually 20- 40 minutes of supplemental instruction. So here we have a situation involving two groups of schools selected because they are high poverty and low achievement. One group is funded through Reading First; the other group is not. The Reading First implementation results in 9 minutes of additional core instruction and there is no mention in the report of the supplemental instruction that should have increased time on task to approximately 120 minutes. Understanding these null results requires a close examination of why the implementation was not more effective in these schools.
Why one would expect 9 additional minutes of instruction to make a significant difference in comprhension among disadvantaged students with significant reading problems is a stretch – not to mention the fact that the schools in the sample were not representative of the many schools that had at a minimum 90 minute blicks for core reading instruction.
I am not sure how the press missed these elementary points that so significantly cloud attempts at interpreting the data. Different agenda I suppose
We shouldn’t allow this single report distract us too much from the bigger picture. The United States Department of Education’s Project Follow Through effort spent a billion dollars over multiple decades examining the impact of different learning strategies on disadvantaged children. The report clearly documented that so called progressive methodologies were damaging to the education of children.
On the implementation side, what are the consequences for the adults in the system for not implementing the program correctly? I’m going to venture to guess that it is something similar to what Hans Blix threatened Kim Jong Il with in the movie Team America- a very stern letter, perhaps? Perhaps not even that?
I’m thinking more of the old Robin Williams routine about police in England not carrying guns – when the criminals run, they yell: “Stop! Or I’ll yell ‘Stop’ again!”
Matthew Ladner raises an interesting point about the comsequences for non-compliane with reading First implementation criteria. What people still do not seem to be able to get their hands around is that Chris Dogherty was thrown to the wolves for trying to ensure that states and local districts adhered to the criteria particularly in relation to the adoption of programs not based on scientific research. Indeed, Chris’s job was to ensure that no Reading First funds be allocated to states/districts that did not meet criteria and he actually rescinded funding when they did not comply with the law. That was his job and it is stated several times in the legislative language. The OIG and the congressional committees set out to destroy him on the premise that he had endorsed programs, but he never did – he just made sure that the states and districts receiving the funds abided by the law. If he did not do this, he would have been breaking the law.
Let’s continue this discussion on Reid Lyon’s new post on this topic, which can be found here: