NAEP Cohort Gains by Scores for State Charter Sectors

December 4, 2017

(Guest Post by Matthew Ladner)

So to get into this chart you had to have a NAEP math score for your charter sector in the 2011 4th grade test, and then again for 8th graders in 2015. Many states either have no charter schools at all, or too few charter schools in 2011 (NC for example), or too few charter schools in 2011 or 2015 to make either sample. Some states fell into this lattercategory despite having venerable charter laws (yes I’m looking at you Indiana and Nevada). If you don’t see your state on the chart, keep calm and open more charter schools.

Otherwise a few notes: Pennsylvania and Maryland both look to have accidentally forgot their charter students any math between 4th grade in 2011 and 8th grade in 2015. I mean there is a few points of gain but one would expect that simply through aging. There might be something odd going on with the sampling or inclusion standards, but if I lived there, I’d be anxious to get to the bottom of it. I don’t so I’m not.

Michigan had the same progress over time as Louisiana despite the fact that Louisiana has been supported with philanthropy and TFA kids to a much larger extent.

Arizona and Colorado are sitting on the bench in the second half eating hot dogs and watching their backups brutalize their hapless opponents.


Math Gains by State Charter Sector

September 9, 2016

(Guest Post by Matthew Ladner)

So just for fun I decided to calculate the cohort math gains for states and state charter sectors in NAEP. Note that on the charter side there are considerable sampling issues in deriving an estimate for a relatively small group of students, making it a really great idea to check a secondary source of data rather than accepting an 8th grade NAEP score for charter students as written on a stone tablet by a higher power.  Various other caveats also apply- high gain scores are not the same as high scores, for instance. The number one gainer below (MN charters) does not have an especially high average 8th grade math score, and in fact lags about ten points below the statewide average for Minnesota. The opposite is true of the number two gainer (AZ charters) which have significantly higher overall scores than their statewide average and high scores overall. Charter sectors dominated by lots of new schools getting their sea legs full of students taking an academic hit getting used to a new school can create an optical illusion in a snapshot, such as those provided by NAEP. Many of these sectors may be on their way to improving in other words as ineffective/undesirable charters close, new ones open and survive, etc.

While NAEP has sampled the same cohort of students as both 4th and 8th graders, they are not of course testing the same students. Students move around, both between states and between district and charter schools. I don’t expect that many states are losing their high performing math students at high rates and having them replaced by low performing math students. In other words, at the state level kids moving around probably does not amount to much because things average out in the aggregate. I’m less confident of this being the case at the charter sector level. There are other caveats that could be dwelt upon, but that would start to violate the Prime Directive.

So okay you’ve been warned- each of these gain scores needs to be viewed in a broader context- far more context than I am going to be able provide here. Having said all of that:


Charter sectors cover both the top and the bottom of the chart- 7 out of the top 10, and the top four overall gains.

Down at the bottom of the chart alas we see the bottom five spots covered by charter sectors. So, Pennsylvania charters, we need to talk.  The NAEP listed your 2011 average 4th grade math score as 241, and your 2015 average 8th grade score as 249. I **ahem** double checked the numbers just to be sure I hadn’t made some mistake. The district numbers for Philadelphia in the TUDA- 225 for 2011 4th graders, 267 for 2015 8th graders.  Both of those scores are catastrophically terrible, but the second one is at least meaningfully higher than the first one. Something goofy with these NAEP numbers? PA charters dominated by dropout recovery programs? Who let the dogs out?

As stated above, no hard and fast conclusions should be drawn from this little insomnia driven exercise, but PA charters might want to turn up the water pressure:

Between the Promise of his greener days and these he masters now

March 21, 2016

(Guest Post by Matthew Ladner)

So I was looking at the National Alliance for Public Charter Schools Dashboard on state charter school sectors trying to figure out how the Arizona Charter sector managed to put it all together to rock the 2015 NAEP like a rock star on an epic hotel room thrashing bender. Just as a reminder, Arizona has long been known as a Wild West of charter schooling- liberal authorizing, relatively light-touch state oversight, etc. Just the sort of thing that a central planner hates. In addition, the sector is not generously funded by national standards, pulling in about $8k per student from all sources, and educates a majority-minority student body. Moreover the Credo analysis from a few years ago held its nose at AZ charter results. And yet, the 2015 NAEP comes in and shows AZ charter students scoring like a New England state in all four exams and these results are backed up by the state AZ Merit exam. What gives?

One possible factor- maturation:

AZ charter sector age

Paul Peterson noted years ago that the Credo analysis of charter test scores had not controlled for two potentially important factors- let’s call them shakedown cruise and transfer effects. Like any social enterprise, a school is unlikely to be at peak effectiveness in the early going.  In 2005-06 Arizona charter schools in years 1-3 outnumbered those that had been operating for 10 or more years three to one. Let the clock run a bit however and by 2013-14 old timers outnumber newbies by more than two to one. As far as transfer effects go, some brand new schools will have nothing but transfers, whereas established schools will typically be breaking in a new class of their youngest grade covered and some transfers. So we waited things out in Arizona and eventually we were rewarded with results like:

8m 2015 AZ Hispanic charter

Well AZ charters- you are growns up, you’re growns up and you’re growns up!

Next factor: churn.

The National Alliance for Public Charter Schools notes that between 2010-11 and 2014-15, 215 new charter schools opened. During the 2010-11 to 2013-14 period 96 Arizona charter schools closed. So factoring in life cycle performance, those 215 that opened may have looked pretty meh when Credo checked in on them as youngsters, but perhaps they were just out having an ale with Falstaff and the gang figuring a few things out getting their sea-legs under them as young organizations.

Let’s assume further that those 96 schools that closed actually were worse than meh on average. Arizona grants 15 year charters, so the oldest of the lot did not start coming up for renewal from the Arizona State Charter School Board until 2010. Charters of course sometimes close due to factors other than state action as well, and academically fantastic schools will be greatly under-represented in this subset. So you open about twice as many schools as you close over the last few years but the ones you open are sorting things out and getting better, while the ones you closed just couldn’t cut it results in:

Florida’s charter sector also rocked the 2015 NAEP. How much did they rock the 2015 NAEP you ask? Well this much:

Hey it’s cute Vermont almost tied Florida charter schools on the reading NAEP, especially since they are 92% Anglo and spend $15,500 per kid. Florida charters btw: 76% minority student body. I’m not going to look up the spending number but it is far less than $15,500. How’s that for closing the achievement gap? If you don’t believe it go and run the numbers for yourself here.

Florida charters seem to be doing something right. Possible maturation factor here as well? I’ll take “Sure Looks that Way to Me” for a thousand Alex:

Florida charter schools age

The National Alliance shows about 300 new charters opening and 101 closing in Florida in recent years. You let parents pick schools that are good fits for their kids, open a new bunch of schools on their way to improving, and close sub-meh, guess what your majority minority sector can rock the NAEP like a New England state.

Now not every charter sector, even among the long-standing ones, sees these kind of results. Many factors can and will influence such scores besides those discussed here. For instance, Michigan charter schools look pretty bad at first blush, but a concentration in Detroit may have something to do with that. The ability to slice and dice the NAEP data has definite limits and multiple factors can be simultaneously at play.  Minnesota however has the nation’s oldest charter law and seems to notably lack the sort of grand-slam ability of the Arizona, Colorado and Florida sectors. A lack of dynamism may be contributing while the average age of Minnesota charter schools has increased, very few new schools have been opening, and very few established schools closing- almost as though charters have their little niche but have been safely contained.

Last month Neerav asked “Is School Supply F***ing Everything?” Maybe so. Keep those new schools coming, let parents sort them out, and if parents don’t torpedo the low-performers don’t renew their charters. Otherwise sit back and enjoy!