(Guest Post by Matthew Ladner)
In the American film classic Animal House there is a scene where students smoke marijuana with their English professor, played by Donald Sutherland, and speculate that it could be the case that the molecules in your fingernails each contain a microscopic universe.
You can’t prove that there aren’t microscopic universes in your fingernails, after all, so they might be in there!
A nice post from Mike Petrilli on the Florida NAEP score gains prompted a response from Kevin Welner from NEPC that shows that the spirit of Sutherland’s Professor Dave Jennings is alive and well at the University of Colorado.
Again there is no attempt to address any of the gaping holes in retention theory. These holes include the fact that Florida’s 4th grade reading scores had improved substantially before the retention policy went into effect, and that they have continued to rise even as retention has fallen off substantially, and that they have fallen off substantially because of a very large improvement in 3rd grade scores.
Welner attempts to tiptoe around this by noting that our EdNext article addressing these points were addressed to a previous Walter Haney paper on the subject rather than the NEPC stuff, which is a distinction without much of a difference. The Chatterji paper contains a carbon copy of the Haney analysis. Amazingly, Chatterji dinged Burke and I for not doing a literature review (not the norm in our tribe) and then cites neither the Education Next paper nor Haney’s analysis. At best, she employed a double standard and at worst, she owes Professor Haney an apology.
Welner’s broader project is to attempt to use the causation problem as a shield. We don’t know, after all, exactly what caused Florida’s remarkable learning gains. Florida’s reformers had to implement their reforms in the real world rather than in a petri dish or in an Intention to Treat Random Assignment study. Welner believes that this allows him the opportunity for strategic nihilism:
The truth might be: (a) there are not actual improvements (the current study is too weak to say whether or not there are), (b) there are improvements, and they’re caused by a combination of all these things, (c) there are improvements, and they’re caused by something none of us pointed to (perhaps the green shirts??), or (d) there are improvements, and they’re caused by some of the things we’re pointing to BUT some of the other things we’re pointing to are actually harming students (just not enough harm to overcome the benefits of the other things).
In other words, when it comes to understanding the FL package of reforms, we are flying blind.
Welner is flying blind all right, but it is by choice. Let’s take each of these little gems on one at a time:
A. The NAEP results show very substantial improvements, as do other indicators.
B. I have always held that the exact cause for the improvement is impossible to know, because Florida’s reformers enacted multiple reforms simultaneously. The logical response to this is not to do none of the Florida reforms, but to do all of them.
C. Florida lurked near the bottom on NAEP for many years, enacted reforms in 1999, and then enjoyed sustained gains over time. While it could be the case that some mysterious X-factor caused the improvement, I’ve yet to hear a plausible theory regarding this. Dan Lips and I addressed multiple possibilities in the Education Next article, including demographic change, spending, etc, and found no evidence to support them.
D. This could be the case, but I haven’t seen a single scrap of evidence to suggest that it is actually the case- return to B above.
Welner is of course correct that there is a correlation and causation problem to consider. As a practical matter, there is nothing else to do but to carefully examine the evidence and history and draw the best conclusions that we can. Dan Lips and I did this in the Education Next article. Florida’s reforms coincided with the student population becoming poorer and less Anglo. State lawmakers increased funding per pupil, but it wasn’t by much and is still below the national average. NEPC complains about a lack of mention of the preschool voucher program when those kids have yet to age into the 4th grade NAEP sample. The class size amendment was implemented very slowly, long after Florida’s scores had begun to rise.
If Dr. Welner would like to provide a plausible explanation for why Florida’s NAEP scores increased so much after 1998, I’d be very interested to read it.
If he prefers to attempt to continue to play games, NEPC’s credibility will go on double secret probation.