J.K. Rowling: The Jeb Bush of the NEPC Florida Fantasy

September 13, 2011

(Guest Post by Matthew Ladner)

The Arizona School Boards Association had their annual law conference last week, and had William Mathis from the Think Tank Review Project present on “Are Things as Sunny as They Seem in Florida?”

I went first, and presented charts like this, showing the vast improvement in Florida’s 3rd grade reading scores:

I have repeatedly asked the Think Tank Review Project people to explain why Florida’s 4th Grade NAEP scores continued to rise in 2007 and 2009 even as 3rd grade retention fell substantially. Or for that matter, why their 3rd grade scores have improved so strongly. Dr. Mathis made no attempt to address the issue.

I also presented charts like these:

Now, call me crazy, but when you are the state called “Arizona” in above chart, you might want to make a careful study of what the other state did to get their English Language Learners to read. This phenomenon  of course is not limited to ELL. Another chart I used showed the combined learning gains on all four NAEP tests for children with disabilities for the entire period we have data from all 50 states (2003-2009).

Just in case you are squinting that’s Florida in red with a gain of 69 points and Arizona in green with a decline of two points.

Dr. Mathis proceeded with his presentation unperturbed. He complained about the 3rd grade retention policy without any effort to explain why Florida’s 3rd grade scores had so profoundly improved, and why Florida’s 4th grade NAEP scores continue to increase even as retention rates have significantly declined.

To give Dr. Mathis’ presentation the fairest possible reading, I would say that he was trying to make the following points: that correlation is not causation, and that to use the terminology of Campbell and Stanley, I had not “controlled for history.” That is to say, there could be other possible explanations for Florida’s gains other than the reforms.

Now it is of course the case that correlation can lead us very much astray, and it is the case that “history” has a nasty habit of bedevilling our theories of causality. As I have noted in the past, however, the Florida reforms unfolded in the real world, rather than in a random assignment study. A great many things unfolded all at once. This is called “life” and there is nothing to be done about this but to gather as much data as possible to draw the best informed decisions we can.

Both Chatteriji and Mathis ignored the Education Next piece in which Dan Lips and I examined other possible explanations for Florida’s gains. Huge spending increases (nope), decline in the percentage of low-income or minority students (nope-increases in both), preschool voucher program (nope- students too young to have aged into the NAEP sample) and class size amendment (nope- implemented very slowly, gains already well under way, formal evaluations negative) and retention law (scores continued to rise even as retention fell). This sort of information might be unhelpful if you are simply trying to get the idea in that something other than a set of hated reforms drove the gains.

Mathis however posited other types of “history” and noted other ways that the world had changed after 1998. On his list of other parts of uncontrolled “history” with regards to Florida’s gains were Harry Potter books (kids reading more fiction) and the more widespread availability of personal computers at home.

Sadly, the format of the panel did not provide time for rebuttal. We had two other people on with us, and took questions from the audience. Had there been such time, however, I would have noted that while Arizona may seem backwards to outsiders (Dr. Mathis lives in Colorado) that we do in fact have Harry Potter books and even personal computers in our humble little patch of cactus. In fact, I am rather confident that Harry Potter books and personal computers became increasingly pervasive in all 50 states.

You never know, Harry Potter books could have powerful educational properties that only manifest themselves on massive peninsulas with high rates of humidity and large concentrations of alligators. The children of Arizona, landlocked in an arid climate, and with not much more in the large lizard department than the occasional Gila monster, may have been left behind. I can’t prove that this isn’t the cause after all.

Nevertheless I’m going to stick with my theory that Governor Bush’s success in implementing a varied and comprehensive set of K-12 reforms in 1999 served as the driver for the large increases in academic attainment seen in Florida’s NAEP scores since 1998. Dr. Mathis and his compatriots can continue to play their stategic nihlism game if they wish, ignoring the problems with their arguments and the studies most on point for the subject at hand (like the regression discontinuity studies of Florida’s retention policy).

Until they put forward a plausible explanation for Florida’s gains, I cannot for the life of me find any reason to take them seriously.

8 Comments | research reports | Tagged: , , , | Permalink
Posted by matthewladner

Ladner and Burke win a Bunkum Award!!!

February 3, 2011

(Guest Post by Matthew Ladner)

The NEA’s “academic” mouthpiece have awarded a Bunkum Award to both me and Lindsey Burke! 

Here it is:

The If I Say It Enough, Will It Still Be Untrue? Award, to the Heritage Foundation’s Closing the Racial Achievement Gap, by Matthew Ladner and Lindsey Burke. The award notes Ladner’s success in repackaging in many different venues and media his spurious claim that a series of Florida reforms, including tax vouchers and grade retention, “caused” racial achievement gaps to narrow in the Sunshine State. “Ladner’s fecundity isn’t really what sets this work apart. It’s his willingness to smash through walls of basic research standards in his dogged pursuit of his policy agenda,” according to our judges. “Nothing in the data or analyses of Dr. Ladner or the Heritage Foundation comes even close to allowing for a causal inference.”
See http://nepc.colorado.edu/thinktank/learning-from-florida

First, I would like to thank the academy, and the Heritage Foundation for giving me a chance to win this wonderful honor.  The scorn of reactionaries is a treasure to cherish. Given that our critic, bless her heart, unknowingly included a table in her report that completely undermined her thesis, I was delighted to see it published.

As to this “inference issue” Dan Lips and I published an article years ago in the nation’s most influential education policy journal examining a number of possible alternate explanations to Florida’s remarkable academic gains. Our critic not only ignored this article, she essentially recreated the argument of another education school professor who we addressed in the piece. She didn’t cite his work either. Oh, and she started her critique off by complaining that Burke and I failed to perform a literature review.

In any case, both Burke and I will have to continue to work hard to earn more of these awards. I hope that we haven’t peaked too early…

14 Comments | vouchers | Tagged: , , , | Permalink
Posted by matthewladner

Burke and Ladner respond to the Think Tank Review Project

December 6, 2010

(Guest Post by Matthew Ladner)

Lindsey Burke and I respond to a critique of our work by the Think Tank Review project. Perhaps my favorite part is where the reviewer chides us for failing to do a literature review but failed to notice that Dan Lips and I had refuted her main contention over a year ago in Education Next. The main point of the review is an attempt to cry foul over Florida’s 4th Grade NAEP scores due to the 3rd grade retention policy.

I’ve changed my mind, this is my favorite part:



This page comes directly from the review. Notice that by the information gathered by the reviewer herself, the percentage of Florida students scoring FCAT 1 (the lowest possible score and the score making a student eligible for retention) in 3rd grade fell from 27% in 2001-02 to 17% in 2008-09. That’s almost a 38% decline.

Likewise, the percentage of African-American students scoring FCAT 1 fell from 41% to 27%, and the Hispanic rate fell from 35% to 21%. Notice that the African-American rate of scoring FCAT 1 now matches the overall rate in 2001-02 had been.

This is called “radical success.”

Notice also that the number of students actually being retained drops by more than half between the first year of the policy and 2008-09. Despite that fact, Florida’s 4th grade NAEP scores continued to climb. If Florida’s NAEP improvement were driven by retention, scores ought to have peaked early in the decade, and then fallen off. Instead, they continued to rise throughout the decade, even as retention declined.

Oh, and Florida’s reading scores improved by almost a grade level before the retention policy even passed. I could go on, point out the practice of mid-year promotions further weakens the “it was all retentions” theory, and/or blather at some length about the regression discontinuity analysis that Jay performed, which strongly points to something other than aging going on with this policy. Click the link if you want to read about it.

The bottom line: these policies worked. The percentage of Florida students scoring below basic on 4th grade reading dropped from 47% to 27% between 1998 and 2009. No one knows exactly how much of which policy moved the needle, but there is a simple solution to this: do all of the policies at the same time.

If Florida lawmakers had mandated in 1999 that students stood off the side of their desks to do jumping jacks to start each school day, and childhood illiteracy dropped like a rock off a cliff, I would be advocating for other states to do the same. At least until such time that someone established that it didn’t add any value.

In some offline conversations I have had with the Think Tank Review people, they seem to think that other states should be “cautious” until we know exactly how much improvement there has been after hair-splitting, and what causes what.

I disagree. In my view, that’s like getting into an argument about whether to use the sprinkler system, the firehose or the buckets of water when kids are running around with their hair on fire. Florida used all the approaches at once, and got great improvement.

Governor-elect Scott seems to busily readying Florida Reform Version 2.0. Somehow I doubt he will be much persuaded by an attempt to muddy the water on Version 1.0.

All is not lost, however.

I will be adding the above table from their study to my Powerpoint, given how well it makes the case for Florida’s reforms.

3 Comments | research reports | Tagged: , , | Permalink
Posted by matthewladner

%d bloggers like this: