(Guest Post by Matthew Ladner)
The Arizona School Boards Association had their annual law conference last week, and had William Mathis from the Think Tank Review Project present on “Are Things as Sunny as They Seem in Florida?”
I went first, and presented charts like this, showing the vast improvement in Florida’s 3rd grade reading scores:
I have repeatedly asked the Think Tank Review Project people to explain why Florida’s 4th Grade NAEP scores continued to rise in 2007 and 2009 even as 3rd grade retention fell substantially. Or for that matter, why their 3rd grade scores have improved so strongly. Dr. Mathis made no attempt to address the issue.
I also presented charts like these:
Now, call me crazy, but when you are the state called “Arizona” in above chart, you might want to make a careful study of what the other state did to get their English Language Learners to read. This phenomenon of course is not limited to ELL. Another chart I used showed the combined learning gains on all four NAEP tests for children with disabilities for the entire period we have data from all 50 states (2003-2009).
Just in case you are squinting that’s Florida in red with a gain of 69 points and Arizona in green with a decline of two points.
Dr. Mathis proceeded with his presentation unperturbed. He complained about the 3rd grade retention policy without any effort to explain why Florida’s 3rd grade scores had so profoundly improved, and why Florida’s 4th grade NAEP scores continue to increase even as retention rates have significantly declined.
To give Dr. Mathis’ presentation the fairest possible reading, I would say that he was trying to make the following points: that correlation is not causation, and that to use the terminology of Campbell and Stanley, I had not “controlled for history.” That is to say, there could be other possible explanations for Florida’s gains other than the reforms.
Now it is of course the case that correlation can lead us very much astray, and it is the case that “history” has a nasty habit of bedevilling our theories of causality. As I have noted in the past, however, the Florida reforms unfolded in the real world, rather than in a random assignment study. A great many things unfolded all at once. This is called “life” and there is nothing to be done about this but to gather as much data as possible to draw the best informed decisions we can.
Both Chatteriji and Mathis ignored the Education Next piece in which Dan Lips and I examined other possible explanations for Florida’s gains. Huge spending increases (nope), decline in the percentage of low-income or minority students (nope-increases in both), preschool voucher program (nope- students too young to have aged into the NAEP sample) and class size amendment (nope- implemented very slowly, gains already well under way, formal evaluations negative) and retention law (scores continued to rise even as retention fell). This sort of information might be unhelpful if you are simply trying to get the idea in that something other than a set of hated reforms drove the gains.
Mathis however posited other types of “history” and noted other ways that the world had changed after 1998. On his list of other parts of uncontrolled “history” with regards to Florida’s gains were Harry Potter books (kids reading more fiction) and the more widespread availability of personal computers at home.
Sadly, the format of the panel did not provide time for rebuttal. We had two other people on with us, and took questions from the audience. Had there been such time, however, I would have noted that while Arizona may seem backwards to outsiders (Dr. Mathis lives in Colorado) that we do in fact have Harry Potter books and even personal computers in our humble little patch of cactus. In fact, I am rather confident that Harry Potter books and personal computers became increasingly pervasive in all 50 states.
You never know, Harry Potter books could have powerful educational properties that only manifest themselves on massive peninsulas with high rates of humidity and large concentrations of alligators. The children of Arizona, landlocked in an arid climate, and with not much more in the large lizard department than the occasional Gila monster, may have been left behind. I can’t prove that this isn’t the cause after all.
Nevertheless I’m going to stick with my theory that Governor Bush’s success in implementing a varied and comprehensive set of K-12 reforms in 1999 served as the driver for the large increases in academic attainment seen in Florida’s NAEP scores since 1998. Dr. Mathis and his compatriots can continue to play their stategic nihlism game if they wish, ignoring the problems with their arguments and the studies most on point for the subject at hand (like the regression discontinuity studies of Florida’s retention policy).
Until they put forward a plausible explanation for Florida’s gains, I cannot for the life of me find any reason to take them seriously.
Jay P. Greene's Blog 
What are the sales rates of Harry Potter books in the two states?
You should start regression-testing every ad hoc theory as they offer them, then toss it out there and say, “nope, what else have you got?”
Not that the theory deserves that treatment, but it would be fun.
I have the flu, and have been “loosing my lunch” all day. That said, it is essential to note that after a decade of accountability in Florida, our 12th graders are doing poorly – and hey – that is the exit year from K-12. For example, in 2005, 35% scored proficient on the NAEP Reading, and in 2009 only 32% scored proficient on the NAEP Reading. How are the students benefiting here? If that is success, then I guess I’ll soon be “loosing my dinner” as well.
Ayn-
No one has ever claimed that Florida is a finished product, South Korea, or education Nirvana. There is no trend data for the 12th grade NAEP, but dropout rates are down, college remediation rates are down, and AP passing scores are up..
If we did have trend data for 12th grade, I suspect that what would find is that things were far more grim in the past.
In Brevard County Florida, remediation rates for Brevard Community College in 1999 showed 59% of their students needed remediation in reading, math or writing. In 2007 those rates rose to 78%. FCAT reading scores for FL 10th graders have been flat since 2001, bobbing up and down from 37% proficiency to 39% proficiency for over a decade.
Extensive research done for the book, Exposing the Public Education System, flat out shows so much statistical data and accounts from all over the country that the bureaucratic education system has failed in its purported mission to educate individual students beyond a shadow of a doubt. Reforms driven from within the existing system cannot deliver success.
So things got worse at one (1) community college in one (1) county, therefore statewide gains . . . don’t count? Didn’t really happen? Help me out here.
You do realize that aggregate gains can occur simultaneously with, or even *cause*, declines in a small subset of the whole?
From a May 2007 OPPAGA Report , a policy report and analysis from Florida: “Over half (55%) of all students entering Florida’s
public postsecondary institutions require remediation in
mathematics, reading, and/or writing. Ninety-four
percent of students who need remediation attend
community colleges.”
A year earlier, another OPPAGA report in April 2006 noted 78% of community college students (on average throughout the state) needed remediation.
I see no updates from the state website. Of course there are outliers that do not “fit” the statistics – but the point is that students are not learning what they need to know prior to graduation. I wish there were better news. Shall we hope for a better report in the near future?
As Matt has already pointed out, the important question is not whether everything is now hunky-dory in Florida (everyone agrees it’s not) but whether Jeb’s reforms made things better. Florida was probably the most abysmal state in the union for education before Jeb, so Jeb’s reforms could easily have made a huge difference and yet still leave lots of room for further progress. To measure these things, you need 1) valid comparisons over time, and 2) data for the whole population. So your college stats leave two open questions: 1) were things even worse than this before Jeb, and 2) could colleges be facing more challenges now because Jeb’s K-12 reforms worked so well that a larger number of difficult-to-educate kids are attending college who pre-Jeb couldn’t do so? I don’t know the answers to those questions, but those are the questions to ask. Meanwhile, Matt’s NAEP data are cross-time and population-wide, so they don’t face these limitations.
[…] responded that critics such as Chatterji and Mathis, a former superintendent of schools in Vermont, ignored his examinations of other possible reasons for student gains in […]