(Guest Post by Matthew Ladner)
The National Center for Education Policy, having put out a “review” whose central thesis was refuted in its own appendix, have decided to try again this time by “reviewing” a powerpoint presentation that Governor Bush made in Michigan. Oh well, I guess it is time to lop off another limb.
Btw, I am not making this business up about them reviewing a powerpoint they didn’t see presented. Perhaps Governor Bush should make sure to spell-check his emails, because they may be up next for a “review.”
I’ll give them credit for improvement: they at least didn’t leave a complete refutation of their own thesis in their own appendix this time. Instead, they simply ignored the fact that their own appendix completely refuted their central thesis last time, and simply restated their central thesis again. From the new review by William Mathis:
Madhabi Chatterji to very likely be the cause for much or most of the NAEP gains—but not in the positive learning sense that Mr. Bush is arguing. Chatterji demonstrates that by screening for low reading scores and then holding these students back a year, the state is able to initially exclude low-scoring students from the fourth-grade NAEP. Then, once these students are promoted to the next grade, the state is able to give the fourth-grade test to a group of students who would otherwise be fifth-graders. That is, these students have another year of learning under their belts. Further, these retained students are disproportionately from minority groups, meaning that the retention policy simultaneously falsely inflates overall scores while creating a misleading impression that the achievement gap is closing.
Ooooops…I still have the Appendix from the previous “review”:
Any of this ringing a bell? Chatterji criticized Burke and me for failing to perform a literature review, then presented Walter Haney’s (flawed) thesis as her own (no citation), and then not only didn’t cite Haney, but also failed to cite or address the refutation of Haney that had been published in the nation’s most influential education policy journal a year earlier. Then, to top it off, she failed to notice that her own appendix undermined her own thesis. Er, I mean Haney’s thesis. Retentions going down, NAEP scores nevertheless going up, 3rd grade scores improving, regression discontinuity evidence…hello?
If retention is causing “much or most” of the improvement in 4th grade scores, why have 3rd grade scores improved so much? Why did reading scores improve by a grade level worth of progress before the retention policy went into effect? Why have scores continued to climb even as retentions have substantially declined?
NEPC response:
The fact that Mathis doesn’t address any of this, but simply reasserts the flawed conclusion (the new reviewer at least attributes it to Chatterji rather than claiming it as his own this time- but should credit Haney instead continuing to rip him off) as valid tells you what you need to know about these guys. They are out to muddy the water if they can, not to engage in a serious debate.
It seems painfully obvious that the reviewer neither watched Governor Bush’s presentation in Michigan, nor even a video of it. Much of the review reads like an ed-school graduate student trying to get their comprehensive exams past a committee including David Berliner and Gene Glass: I cited you! I cited you! The poor chap seems to think that things these guys wrote in the past about programs in other states serves as proof positive about programs in Florida (they don’t) and that a consensus among left wing ed school profs constitutes evidence (it doesn’t).
“Unfortunately, if research is our guide, the effect of the Florida reforms will likely prove to be a more inequitable and inadequate educational system,” Mathis wrote. Mathis should have said “Unfortunately, if the nonsense that passes for ‘research’ in my ideological tribe is our guide, the effect of the Florida reforms will likely prove to be a more inequitable and inadequate educational system.”
That’s an awfully tart statement. You were just thinking “Can he back that up with evidence?” Glad you asked!
It just so happens that I have been digging into the NAEP data to look at achievement gap trends by state. I combined all four major NAEP exams (4th grade reading, 4th grade math, 8th grade reading, 8th grade math) for the entire period that all 50 states participated (2003-2009). Anyone can go and look these numbers up for themselves, and here is a little sneak peek:
If you guessed that Florida made more progress than any other state in narrowing the Black-White achievement gap on the combined NAEP exams, give yourself a gold star. If you don’t believe it, go look the numbers up for yourself. White students made gains, but Black students made bigger gains. This is really the only good way to narrow an achievement gap, and it is the way it happened in Florida. The same is true of the Hispanic-White gap- Florida led the nation, and bettered the national average by a factor of almost six. Florida achieved the second largest narrowing in the gap between poor and non-poor students, and between children with disabilities and without them.
However, following the example of Arthur, King of the Britons, we’ll call it a draw.
UPDATE Commentor Chan S detected a computational error in the Black-White achievement gap which underestimated Florida’s progress in reducing the gap. After double checking the figures, I’ve included a corrected chart.
Jay P. Greene's Blog 
WOW, Jeb Bush, another Bush RINO. Who’d have guessed?
Mom w/b-
Where is the common core vent? Did someone kidnap your handle, or have you turned over a new leaf?
-ML
Could you share the achievement gap measure used (average scale scores, percentages below basic, percentages at or above proficient, other)? Thanks very much.
Average scale scores, so in the chart above measuring the Black-White achievement gap, I took all four tests in 2003, combined them, and subtracted the average score for Black students from the average score for White students.
I did the process again in 2009 and then compared the 2003 meta gap to the 2009 meta gap.
Thanks! Maybe I’m misapplying the formula, but I get -23.5 for Florida and -1 for Wisconsin using the whole number average scale scores (the double digit “declining gap” number for Wisconsin in the chart is what prompted my query).
Chan-
This is precisely why it is a good idea for me to test drive things on the blog. I found a computational error that underestimated Florida’s progress in narrowing the Black-White achievement gap, and overestimated Wisconsin’s progress. Florida still comes in first overall, just by a wider margin this time.
I’m going to triple check the numbers, and then correct the table. Thanks for the fact check!
-ML
I am surprised that this article – and most commentary about florida’s miracle naep scores – fail to take into account the huge change in the racial makeup of the 4th grade test taking pool. As a result of the end of social promotion in Florida – most blacks and Hispanics were held back In the third grade, thus dramatically improving the demographics of the test takers in the fourth grade. To put it bluntly, they held back the dumb kids, and voila, the test scores increased. Are we really surprised?
Ummm,,,Jerry, I think you missed the point of the article.
Take a look at Table 1 from the appendix. The total number of students retained drops by more than half between 2002 and 2008. Meanwhile Florida’s NAEP scores continue to increase.
Then look at the percentage of Black students scoring FCAT 1 in 3rd grade (the retention qualifying score). Starts at 41% the year before the policy, drops to 27% by 2008-09.
Hispanics drop from 35% FCAT 1 to 21% FCAT 1 in 3rd grade.
This “they held back the dumb kids, and voila, the test scores increased” theory cannot explain why Florida’s 4th grade NAEP scores continue to improve even as retention substantially falls.
not so fast, Matthew. Take a look at the numbers of students in various grades in Florida during these miracle times. From 2002-03 to 2004-05 and again the following year, the number of third graders was roughly 10 percent higher than those in the second grade. Then, between third and fourth grades, enrollment suddenly shrank a comparable amount. In 2006, that’s when Florida started flunking large numbers of third graders. Because of the new retention policy, low-achieving third graders were still in third grade when the NAEP 2005 fourth grade math test was given. With only the higher-achieving students taking the test, the scores jumped. What’s more, the state flunked a much higher proportion of Black than White students—no wonder the achievement gap shrank.
I’d like to attribute these gains to the fact that schools were now being given letter grades, and various other Jeb Bush reforms (most of which I agree with) but I really think the 4th Grade NAEP scores had more to do with the composition of the 4th grader class than any other factor…..
Jerry-
The 3rd grade retention policy went into effect in 2002-03, and there were more than twice as many students retained that year than in 2008-09.
If retention is driving the bus here, can you explain why why Florida’s 4th graders scored 7 points higher in reading on the 2009 NAEP than the 2005 group, which would have clearly been far more influenced by the retention policy. Remember that 10 points roughly equals an average grade level gain, so 7 points is meaningful increase.