(Guest Post by Matthew Ladner)
The news overall is grim. You want to be in the top right quadrant of this chart. Some states thankfully did land there-including the state with the largest student population- but many only barely due to small reading gains. If you need the dot size to push you in, it doesn’t count-better luck next NAEP.
You don’t want to land in the top left or in the bottom right, and you most of all don’t want to see your state in the bottom left quadrant (declines in both subjects). Mike Petrilli used the phrase “lost decade” to describe the results. Some states seem far more lost than others, but it is hard to find fault with that assessment overall.
The two main reform strategies employed since the 1990s have involved test-based accountability and increased parental choice. During the era covered by the top chart, the test-based folks swung for the fences by creating a federal incentive for states to adopt a preferred set of academic standards and to pass statewide teacher evaluation systems based upon the scores on those tests. Gigantic investments of political and financial capital supported these policy changes, but it is hard to characterize the results as much more than disappointing.
Now some of you will be thinking around about now “oh yeah but we’ve expanded choice during this period as well!” That is true, and while we have numerous studies establishing positive competitive effects on district schools from choice programs, few states have choice programs going at a scale to place a large amount of pressure on district enrolments. NAEP does however allow us to track state charter sector gains over time. Sixteen state charter sectors had scores for 8th grade math and reading in both 2009 and 2017, allowing the following calculation:
Excel had to change the scale of the axis for the above chart. You may not have noticed. Putting state averages and state charter sector averages into the same chart will help:
Suddenly those statewide gains in Arizona, California and Mississippi (i.e. the good ones) from the first chart don’t seem so impressive eh? I’m thinking out loud here and inviting you along for the ride. Gains aren’t everything, so the next iteration will include achievement and gains by subject area, but for the huge gainer sectors (spoiler alert) they didn’t get that way with low 2017 scores. I could go on about standard errors being bigger for charter sectors and whatnot, but who are you going to believe a boring statistics lecture or your own lying eyes? If someone can explain why random error would systematically dramatically favor charter sectors, I’m all ears and the comment section eagerly awaits your thoughtful challenge.
In fact there is a white lie in the above chart-some of the sectors and states that look meh in this chart immediately above had very high scores in both 2009 and 2017, and while not ideal there is no crime in holding your mud with high scores. Last year my friend Robert Pondiscio convinced me that combining achievement and gains to provide a clearer picture, so here goes for 8th grade math:
In this chart you either want to have large gains, or high scores, or preferably both of these things. I’m for instance not inclined to criticize Idaho charter schools for modest gains given that they outscore Massachusetts and all despite spending about half the amount per pupil.
Well…yup it is officially time:
Jay P. Greene's Blog 
Do you have data to control for student factors in the charter sector a la Urban Institute tool? Looking at Arizona’s charter sector, it has to be wealthier than the state with BASIS and GreatHearts, right?
I’m guessing that Basis and GH educate about 15% of AZ charter students, and both operate high FRL campuses these days. FRL numbers are generally unreliable around the country and are ultra-unreliable in the AZ charter sector due to a large number of non-participating schools. I can tell you that Arizona charter schools educate a majority-minority student population. I’ve checked on differences in SPED rates in state testing data, and the difference is not nearly as large as is often imagined.
Having said all of that I don’t doubt that differences in student populations explain some of the differences in performance between AZ charters and AZ districts. This is the sort of question that can’t be answered definitively without a random assignment study. Ultimately what I hope to see is both AZ districts and charters continue to make academic progress.
Thanks for the response, very helpful. One follow up: do we have data through NAEP demonstrating if/how the demographics of the charter sector participants have changed over the 09-17 window? I’m assuming that state populations have negligibly changed over that time, but the ‘09 sector and ‘17 sector could theoretically be quite different.
No grounds for assuming state population changes are negligible. Ten years is a long time.
Limited info on AZ but trending less Anglo over time.