J.K. Rowling: The Jeb Bush of the NEPC Florida Fantasy

September 13, 2011

(Guest Post by Matthew Ladner)

The Arizona School Boards Association had their annual law conference last week, and had William Mathis from the Think Tank Review Project present on “Are Things as Sunny as They Seem in Florida?”

I went first, and presented charts like this, showing the vast improvement in Florida’s 3rd grade reading scores:

I have repeatedly asked the Think Tank Review Project people to explain why Florida’s 4th Grade NAEP scores continued to rise in 2007 and 2009 even as 3rd grade retention fell substantially. Or for that matter, why their 3rd grade scores have improved so strongly. Dr. Mathis made no attempt to address the issue.

I also presented charts like these:

Now, call me crazy, but when you are the state called “Arizona” in above chart, you might want to make a careful study of what the other state did to get their English Language Learners to read. This phenomenon  of course is not limited to ELL. Another chart I used showed the combined learning gains on all four NAEP tests for children with disabilities for the entire period we have data from all 50 states (2003-2009).

Just in case you are squinting that’s Florida in red with a gain of 69 points and Arizona in green with a decline of two points.

Dr. Mathis proceeded with his presentation unperturbed. He complained about the 3rd grade retention policy without any effort to explain why Florida’s 3rd grade scores had so profoundly improved, and why Florida’s 4th grade NAEP scores continue to increase even as retention rates have significantly declined.

To give Dr. Mathis’ presentation the fairest possible reading, I would say that he was trying to make the following points: that correlation is not causation, and that to use the terminology of Campbell and Stanley, I had not “controlled for history.” That is to say, there could be other possible explanations for Florida’s gains other than the reforms.

Now it is of course the case that correlation can lead us very much astray, and it is the case that “history” has a nasty habit of bedevilling our theories of causality. As I have noted in the past, however, the Florida reforms unfolded in the real world, rather than in a random assignment study. A great many things unfolded all at once. This is called “life” and there is nothing to be done about this but to gather as much data as possible to draw the best informed decisions we can.

Both Chatteriji and Mathis ignored the Education Next piece in which Dan Lips and I examined other possible explanations for Florida’s gains. Huge spending increases (nope), decline in the percentage of low-income or minority students (nope-increases in both), preschool voucher program (nope- students too young to have aged into the NAEP sample) and class size amendment (nope- implemented very slowly, gains already well under way, formal evaluations negative) and retention law (scores continued to rise even as retention fell). This sort of information might be unhelpful if you are simply trying to get the idea in that something other than a set of hated reforms drove the gains.

Mathis however posited other types of “history” and noted other ways that the world had changed after 1998. On his list of other parts of uncontrolled “history” with regards to Florida’s gains were Harry Potter books (kids reading more fiction) and the more widespread availability of personal computers at home.

Sadly, the format of the panel did not provide time for rebuttal. We had two other people on with us, and took questions from the audience. Had there been such time, however, I would have noted that while Arizona may seem backwards to outsiders (Dr. Mathis lives in Colorado) that we do in fact have Harry Potter books and even personal computers in our humble little patch of cactus. In fact, I am rather confident that Harry Potter books and personal computers became increasingly pervasive in all 50 states.

You never know, Harry Potter books could have powerful educational properties that only manifest themselves on massive peninsulas with high rates of humidity and large concentrations of alligators. The children of Arizona, landlocked in an arid climate, and with not much more in the large lizard department than the occasional Gila monster, may have been left behind. I can’t prove that this isn’t the cause after all.

Nevertheless I’m going to stick with my theory that Governor Bush’s success in implementing a varied and comprehensive set of K-12 reforms in 1999 served as the driver for the large increases in academic attainment seen in Florida’s NAEP scores since 1998. Dr. Mathis and his compatriots can continue to play their stategic nihlism game if they wish, ignoring the problems with their arguments and the studies most on point for the subject at hand (like the regression discontinuity studies of Florida’s retention policy).

Until they put forward a plausible explanation for Florida’s gains, I cannot for the life of me find any reason to take them seriously.

8 Comments | research reports | Tagged: , , , | Permalink
Posted by matthewladner

It’s just a flesh wound!

July 5, 2011

(Guest Post by Matthew Ladner)

The National Center for Education Policy, having put out a “review” whose central thesis was refuted in its own appendix, have decided to try again this time by “reviewing” a powerpoint presentation that Governor Bush made in Michigan. Oh well, I guess it is time to lop off another limb.

Btw, I am not making this business up about them reviewing a powerpoint they didn’t see presented. Perhaps Governor Bush should make sure to spell-check his emails, because they may be up next for a “review.”

I’ll give them credit for improvement: they at least didn’t leave a complete refutation of their own thesis in their own appendix this time. Instead, they simply ignored the fact that their own appendix completely refuted their central thesis last time, and simply restated their central thesis again. From the new review by William Mathis:

Madhabi Chatterji to very likely be the cause for much or most of the NAEP gains—but not in the positive learning sense that Mr. Bush is arguing. Chatterji demonstrates that by screening for low reading scores and then holding these students back a year, the state is able to initially exclude low-scoring students from the fourth-grade NAEP. Then, once these students are promoted to the next grade, the state is able to give the fourth-grade test to a group of students who would otherwise be fifth-graders. That is, these students have another year of learning under their belts. Further, these retained students are disproportionately from minority groups, meaning that the retention policy simultaneously falsely inflates overall scores while creating a misleading impression that the achievement gap is closing.

Ooooops…I still have the Appendix from the previous “review”:

Any of this ringing a bell? Chatterji criticized Burke and me for failing to perform a literature review, then presented Walter Haney’s (flawed) thesis as her own (no citation), and then not only didn’t cite Haney, but also failed to cite or address the refutation of Haney that had been published in the nation’s most influential education policy journal a year earlier. Then, to top it off, she failed to notice that her own appendix undermined her own thesis. Er, I mean Haney’s thesis. Retentions going down, NAEP scores nevertheless going up, 3rd grade scores improving, regression discontinuity evidence…hello?

If retention is causing “much or most” of the improvement in 4th grade scores, why have 3rd grade scores improved so much? Why did reading scores improve by a grade level worth of progress before the retention policy went into effect? Why have scores continued to climb even as retentions have substantially declined?

NEPC response:

The fact that Mathis doesn’t address any of this, but simply reasserts the flawed conclusion (the new reviewer at least attributes it to Chatterji rather than claiming it as his own this time- but should credit Haney instead continuing to rip him off) as valid tells you what you need to know about these guys. They are out to muddy the water if they can, not to engage in a serious debate.

It seems painfully obvious that the reviewer neither watched Governor Bush’s presentation in Michigan, nor even a video of it. Much of the review reads like an ed-school graduate student trying to get their comprehensive exams past a committee including David Berliner and Gene Glass: I cited you! I cited you! The poor chap seems to think that things these guys wrote in the past about programs in other states serves as proof positive about programs in Florida (they don’t) and that a consensus among left wing ed school profs constitutes evidence (it doesn’t).

“Unfortunately, if research is our guide, the effect of the Florida reforms will likely prove to be a more inequitable and inadequate educational system,” Mathis wrote. Mathis should have said “Unfortunately, if the nonsense that passes for ‘research’ in my ideological tribe is our guide, the effect of the Florida reforms will likely prove to be a more inequitable and inadequate educational system.”

That’s an awfully tart statement. You were just thinking “Can he back that up with evidence?” Glad you asked!

It just so happens that I have been digging into the NAEP data to look at achievement gap trends by state. I combined all four major NAEP exams (4th grade reading, 4th grade math, 8th grade reading, 8th grade math) for the entire period that all 50 states participated (2003-2009). Anyone can go and look these numbers up for themselves, and here is a little sneak peek:

If you guessed that Florida made more progress than any other state in narrowing the Black-White achievement gap on the combined NAEP exams, give yourself a gold star. If you don’t believe it, go look the numbers up for yourself. White students made gains, but Black students made bigger gains. This is really the only good way to narrow an achievement gap, and it is the way it happened in Florida. The same is true of the Hispanic-White gap- Florida led the nation, and bettered the national average by a factor of almost six. Florida achieved the second largest narrowing in the gap between poor and non-poor students, and between children with disabilities and without them.

However, following the example of Arthur, King of the Britons, we’ll call it a draw.

UPDATE Commentor Chan S detected a computational error in the Black-White achievement gap which underestimated Florida’s progress in reducing the gap. After double checking the figures, I’ve included a corrected chart.

10 Comments | teacher union | Tagged: , , , | Permalink
Posted by matthewladner

%d bloggers like this: