Cohort NAEP Gains by Spending

October 11, 2017


(Guest Post by Matthew Ladner)

So I’ve been cooking up some new charts on NAEP statewide cohort gains for math and reading by state K-12 per pupil spending trends. Imo the cohort gains are pretty good overall measure of statewide school quality, albeit not a perfect one. Student demographics influence all scores, but if school quality is going to assert itself they should have less of an influence on 8th grade scores than 4th grade scores simply because the kids have been in school longer. Thus question addressed along the horizon in this chart is how much math did your state’s students learn between 4th grade in 2009 and 8th grade in 2013? This is plotted against the trend in per capita spending between 2007 and 2014 per NCES.

So let’s note a few things here. First once again there is a lack of a discernible relationship between spending trend and academic outcomes. You had some states that made bid increases that bombed, and others that made big cuts and lead the nation in gains (take a bow Arizona educators and policymakers!)

Maybe this was a fluke. What happens if you do it again for math cohort gains between 2011 4th graders and 2015 8th graders?

Using my Professor X mutant super-power, I am reading your thoughts. You were thinking “Okay Ladner we get it something good is going in math. What about other subjects?” Fair enough- we can only do cohort gains in math and reading, so here is the cohort reading gains:

Well would you look at that- tied for second. Cohort gains are one method for measuring gains, but we can also look at over time gains for different cohorts of students, which allows us to bring in 4th and 8th grade science. Here is what those look like for the entire period we can get readings on all six tests (new results will be released in January 2017):

Can Arizona keep it up? I certainly hope so and we will find out in January.


The Origin of Arizona’s Nation Leading NAEP gains

May 1, 2017

(Guest Post by Matthew Ladner)

It’s been interesting for me to watch different columnists at the Arizona Republic react differently to the news that Arizona is the only state that has been the only state with statistically significant gains on all six NAEP exams. Bob Robb stated in a recent column that he supports choice but thinks it is limited strategy for improving outcomes. His colleague Joanna Allhands noted the fact that Arizona has lead the nation in NAEP gains, but said we have no idea of why that was the case.

I hope the above chart explains why I think even Robb is selling choice short, why Allhands should reconsider K-12 agnosticism. Formally Allhands is correct that we do not know why Arizona has been leading the nation in gains, but I hope a close examination of the above chart will be fairly persuasive that Arizona’s choice policies had a great deal to do with it.

So let’s peel the above chart like an onion. The first layer- blue columns- are the national public school gains across all six NAEP subjects (4th and 8th grade Math, Reading and Science). These are across time gains rather than cohort gains, calculated by subtracting the 2009 NAEP score from the 2015 score. Looking at the blue columns shows that the national progress falls into the strictly meh: 1 point on 4th grade math, -1 point on 8th grade math, one point on 4th grade reading, 2 points on 8th grade reading, four points on 4th and 8th grade science. Nothing much to celebrate nationally.

Next look at the yellow columns- these are the 2009 to 2015 gains for Arizona school districts (no charters). As you can see, these gains are consistently larger than the blue national public numbers, especially in math and science.

Third, look at the total statewide gains (Arizona flag columns). These are the gains for the combined district and charter schools between 2009 and 2015. As you can see, these gains are consistently larger than the district gains alone (yellow columns) and far, far larger than the national public averages (blue columns). Arizona was the only state to have statistically significant gains on all six NAEP exams between 2009 and 2015.

Finally, in the back in red tower the gains for Arizona charter schools between 2009 and 2015. The over/under for percentage of Arizona students attending charters in 2015 was around 15%, so although these gains are huge, they directly move the statewide needle by the differences between the yellow district columns and the flag columns. *See boring stat nerd note below.

The above gains represent the 2015 minus the 2009 scores-for example Arizona’s 8th grade math score minus Arizona’s 2009 8th grade math scores. A different method for calculating NAEP gains is to follow the progress of a single cohort of students across time. The NAEP math and reading tests have been timed and scaled to allow such comparisons- 4th graders took for instance the 4th grade NAEP math in 2011 and the NAEP 8th grade math exam in 2015. So…which state’s students learned the most about math between their 4th grade test in 2011 and and the 8th grade test in 2015?

Nationally American students gained 41 points between the 2011 4th grade exam and the 2015 8th grade exam-so nationally about 10 points per year. Arizona lead the nation with a 48 point gain. So how did Arizona charter schools fare in this comparison? **See second nerdy statistical note.

Note that the gap between Arizona charters and districts in cohort gains (12 points) is almost as large as the gap between gain leading Arizona and the lowest rated state (Alabama). So what does this mean in practical terms? The faster rate of improvement meant that Arizona charter school students got to do this on the 2015 NAEP, which is pretty cool if you like majority-minority schooling sectors that show globally competitive levels of academic achievement:

Finally, we have a rich set of empirical studies that suggest that parental choice leads to academic gains in traditional district school systems. Going back to the first slide, we have reason to suspect that some of the differences between the yellow and the blue columns relates to parental choice. If you suspect that budget cuts lead to academic gains (I don’t) then okay maybe, or if you can come up with a reason why new standards would have a very unusually large positive impact in Arizona when they flopped around the country, I’m willing to entertain a story to that effect, but it sounds like an implausibly complicated story.

On the choice side, round about 2007, the economy collapsed in a way that made a lot of property available, and Arizona’s charter sector put the peddle to the metal. Arizona charter school sector rose by a rousing 43,000 students in 2013 for example. The number of students exercising private choice also increased during this period, and statewide enrollment growth slowed, but that increase pales next to that of Arizona charters, which increased from 95,000ish students in 2008-09 to 188,000ish students in 2013-14.

We can feel confident that some of the difference between the statewide numbers and the blue columns relates to parental choice. We can feel very confident indeed that some of the difference between the red columns and the blue columns is related to parental choice. I’m open to other interpretations-and the comment section is open-but Occam’s Razor leads me to believe that a huge increase in the prevalence of parental choice that occurred during the Great Recession lead to direct benefits (high charter school scores) and powerful competitive effects (attract students or suffer real consequences- real accountability as opposed to the phony slap on the hand sort).

*The 2009 NAEP Arizona charter school estimates had large standard errors of estimates, owing to the considerably smaller size of the sector at the time. Unless Arizona charter school sleeper agents have infiltrated the NAEP there is little reason to suspect that random error will consistently advantage charter schools across six NAEP exams. Random error in both the 2015 and 2009 estimates means that the red columns in the first chart could be either smaller or bigger if we had actually tested everyone, but I’m at a loss for a reason to think of a reason why the errors across twelve different testing samples (six in 2009 and another six in 2015) would consistently line up to produce a mirage of Arizona charter school academic conquest, again absent sleeper agent infiltration.

** Standard error plays into the calculation of cohort gains as well on both ends of the calculation (in this case 2011 and 2015 scores) such that either could have been higher or lower if we had tested the entire population. Standard errors are larger for sub-population estimates than statewide averages, but again could play either way. For example if the population score for 2011 charter school students was higher than the NAEP estimate, the cohort gain will be overestimated, and if the true population score in 2011 was actually lower, then the cohort gain reported here would be an underestimate. All NAEP scores are estimates based on samples. Arizona’s charter school students displayed larger than any state cohort gains than any other state in both math and reading, but we cannot have the same level of confidence in these estimates as in statewide averages. Again, assuming random error and a lack of Arizona charter school sleeper agents in NAEP, we would not expect random error to consistently advantage Arizona charter schools.

Finally, the state’s AZ Merit exam also shows large advantages for Arizona charter school students vis a vis district students. Sampling is not an issue in AZ Merit, and these results lend reinforcement to the NAEP results. Unless…AZ charter school sleeper agents infiltrated the state’s testing system as well…




Arizona Leads the Nation in NAEP Gains-Now It is Time to Go from Good to Great

July 14, 2016

(Guest Post by Matthew Ladner)

For you incurable skeptics, here are a few charts on Arizona’s glorious six-year (and counting?) reign as the NAEP gain champion, starting with 2009 to 2013:

Math NAEP gains 2009 to 2013

So just to provide some play by play here- Arizona 4th graders were 9 points below the national average on 4th grade math in 2009, but when this cohort reached 8th grade in 2013, they were only 4 points below the national average- within striking distance of the national average.

Here is what happened in the next cohort we can track in NAEP math- 4th graders in 2011 to 8th graders in 2015.

NAEP Math Cohort gain 2015

So in 2011, Arizona 4th graders scored five points below the national average on 4th grade math, still ranked in the bottom 10 of states. In 2015 the same cohort of students scored 2 points above the national average on 8th grade math. This was the first time in the history of the NAEP that Arizona had scored above the national average in any NAEP exam in any subject.

So where does this leave Arizona? Unfinished to be sure, but headed in the right direction. NAEP 8th grade scores are more reflective of the overall quality of a school system than 4th grade scores in my opinion, as you have additional years of schooling. In 2015, Arizona had moved to within the margin of error of the national average on both NAEP math and reading (slightly above in Math, even more slightly below in Reading). Arizona students were below the national average and outside the margin of error on both Math and Reading in 2009.

Of course the national average itself is a milestone but not a resting place- the United States does not rank well against other countries. We need to keep it up. Arizona was never going to become even an average performing state without above average gains, and no other state matches the gains seen by Arizona students over the last six years- so a bit of celebration is in order:

Guess which state had the largest overall NAEP gains in 2013…

July 13, 2016

(Guest Post by Matthew Ladner)

I’ve already been happy to report that measuring NAEP gains between 4th grade scores in 2011 and 8th grade scores in 2015, Arizona banked the largest overall gain in achievement. Just out of curiosity I decided to rank the states during the previous period (2009 to 2013) the same way- 4th grade scores in 2009 compared to 8th grade scores in 2013. I did this separately for Math and Reading.

Any guesses on which state came in first in overall gains? Here is a few clues:

Arizona wins again! First in overall math gains, fifth in overall reading gains, highest overall combined ranking. Feel free to pop out and take notes- we’ve been doing this since (at least) 2009. Yes, feel free to bring your golf clubs.

Oh yes- I almost forgot: