Jay P. Greene's Blog

(Guest Post by Matthew Ladner)

The news overall is grim. You want to be in the top right quadrant of this chart. Some states thankfully did land there-including the state with the largest student population- but many only barely due to small reading gains. If you need the dot size to push you in, it doesn’t count-better luck next NAEP.

You don’t want to land in the top left or in the bottom right, and you most of all don’t want to see your state in the bottom left quadrant (declines in both subjects). Mike Petrilli used the phrase “lost decade” to describe the results. Some states seem far more lost than others, but it is hard to find fault with that assessment overall.

The two main reform strategies employed since the 1990s have involved test-based accountability and increased parental choice. During the era covered by the top chart, the test-based folks swung for the fences by creating a federal incentive for states to adopt a preferred set of academic standards and to pass statewide teacher evaluation systems based upon the scores on those tests. Gigantic investments of political and financial capital supported these policy changes, but it is hard to characterize the results as much more than disappointing.

Now some of you will be thinking around about now “oh yeah but we’ve expanded choice during this period as well!” That is true, and while we have numerous studies establishing positive competitive effects on district schools from choice programs, few states have choice programs going at a scale to place a large amount of pressure on district enrolments. NAEP does however allow us to track state charter sector gains over time. Sixteen state charter sectors had scores for 8th grade math and reading in both 2009 and 2017, allowing the following calculation:

Excel had to change the scale of the axis for the above chart. You may not have noticed. Putting state averages and state charter sector averages into the same chart will help:

Suddenly those statewide gains in Arizona, California and Mississippi (i.e. the good ones) from the first chart don’t seem so impressive eh? I’m thinking out loud here and inviting you along for the ride. Gains aren’t everything, so the next iteration will include achievement and gains by subject area, but for the huge gainer sectors (spoiler alert) they didn’t get that way with low 2017 scores. I could go on about standard errors being bigger for charter sectors and whatnot, but who are you going to believe a boring statistics lecture or your own lying eyes? If someone can explain why random error would systematically dramatically favor charter sectors, I’m all ears and the comment section eagerly awaits your thoughtful challenge.

In fact there is a white lie in the above chart-some of the sectors and states that look meh in this chart immediately above had very high scores in both 2009 and 2017, and while not ideal there is no crime in holding your mud with high scores. Last year my friend Robert Pondiscio convinced me that combining achievement and gains to provide a clearer picture, so here goes for 8th grade math:

In this chart you either want to have large gains, or high scores, or preferably both of these things. I’m for instance not inclined to criticize Idaho charter schools for modest gains given that they outscore Massachusetts and all despite spending about half the amount per pupil.

Well…yup it is officially time: