Jay P. Greene's Blog

(Guest Post by Matthew Ladner)

The National Center for Education Policy, having put out a “review” whose central thesis was refuted in its own appendix, have decided to try again this time by “reviewing” a powerpoint presentation that Governor Bush made in Michigan. Oh well, I guess it is time to lop off another limb.

Btw, I am not making this business up about them reviewing a powerpoint they didn’t see presented. Perhaps Governor Bush should make sure to spell-check his emails, because they may be up next for a “review.”

I’ll give them credit for improvement: they at least didn’t leave a complete refutation of their own thesis in their own appendix this time. Instead, they simply ignored the fact that their own appendix completely refuted their central thesis last time, and simply restated their central thesis again. From the new review by William Mathis:

Madhabi Chatterji to very likely be the cause for much or most of the NAEP gains—but not in the positive learning sense that Mr. Bush is arguing. Chatterji demonstrates that by screening for low reading scores and then holding these students back a year, the state is able to initially exclude low-scoring students from the fourth-grade NAEP. Then, once these students are promoted to the next grade, the state is able to give the fourth-grade test to a group of students who would otherwise be fifth-graders. That is, these students have another year of learning under their belts. Further, these retained students are disproportionately from minority groups, meaning that the retention policy simultaneously falsely inflates overall scores while creating a misleading impression that the achievement gap is closing.

Ooooops…I still have the Appendix from the previous “review”:

Any of this ringing a bell? Chatterji criticized Burke and me for failing to perform a literature review, then presented Walter Haney’s (flawed) thesis as her own (no citation), and then not only didn’t cite Haney, but also failed to cite or address the refutation of Haney that had been published in the nation’s most influential education policy journal a year earlier. Then, to top it off, she failed to notice that her own appendix undermined her own thesis. Er, I mean Haney’s thesis. Retentions going down, NAEP scores nevertheless going up, 3rd grade scores improving, regression discontinuity evidence…hello?

If retention is causing “much or most” of the improvement in 4th grade scores, why have 3rd grade scores improved so much? Why did reading scores improve by a grade level worth of progress before the retention policy went into effect? Why have scores continued to climb even as retentions have substantially declined?

NEPC response:

The fact that Mathis doesn’t address any of this, but simply reasserts the flawed conclusion (the new reviewer at least attributes it to Chatterji rather than claiming it as his own this time- but should credit Haney instead continuing to rip him off) as valid tells you what you need to know about these guys. They are out to muddy the water if they can, not to engage in a serious debate.

It seems painfully obvious that the reviewer neither watched Governor Bush’s presentation in Michigan, nor even a video of it. Much of the review reads like an ed-school graduate student trying to get their comprehensive exams past a committee including David Berliner and Gene Glass: I cited you! I cited you! The poor chap seems to think that things these guys wrote in the past about programs in other states serves as proof positive about programs in Florida (they don’t) and that a consensus among left wing ed school profs constitutes evidence (it doesn’t).

“Unfortunately, if research is our guide, the effect of the Florida reforms will likely prove to be a more inequitable and inadequate educational system,” Mathis wrote. Mathis should have said “Unfortunately, if the nonsense that passes for ‘research’ in my ideological tribe is our guide, the effect of the Florida reforms will likely prove to be a more inequitable and inadequate educational system.”

That’s an awfully tart statement. You were just thinking “Can he back that up with evidence?” Glad you asked!

It just so happens that I have been digging into the NAEP data to look at achievement gap trends by state. I combined all four major NAEP exams (4th grade reading, 4th grade math, 8th grade reading, 8th grade math) for the entire period that all 50 states participated (2003-2009). Anyone can go and look these numbers up for themselves, and here is a little sneak peek:

If you guessed that Florida made more progress than any other state in narrowing the Black-White achievement gap on the combined NAEP exams, give yourself a gold star. If you don’t believe it, go look the numbers up for yourself. White students made gains, but Black students made bigger gains. This is really the only good way to narrow an achievement gap, and it is the way it happened in Florida. The same is true of the Hispanic-White gap- Florida led the nation, and bettered the national average by a factor of almost six. Florida achieved the second largest narrowing in the gap between poor and non-poor students, and between children with disabilities and without them.

However, following the example of Arthur, King of the Britons, we’ll call it a draw.

UPDATE Commentor Chan S detected a computational error in the Black-White achievement gap which underestimated Florida’s progress in reducing the gap. After double checking the figures, I’ve included a corrected chart.