(Guest Post by Matthew Ladner)
It’s been interesting for me to watch different columnists at the Arizona Republic react differently to the news that Arizona is the only state that has been the only state with statistically significant gains on all six NAEP exams. Bob Robb stated in a recent column that he supports choice but thinks it is limited strategy for improving outcomes. His colleague Joanna Allhands noted the fact that Arizona has lead the nation in NAEP gains, but said we have no idea of why that was the case.
I hope the above chart explains why I think even Robb is selling choice short, why Allhands should reconsider K-12 agnosticism. Formally Allhands is correct that we do not know why Arizona has been leading the nation in gains, but I hope a close examination of the above chart will be fairly persuasive that Arizona’s choice policies had a great deal to do with it.
So let’s peel the above chart like an onion. The first layer- blue columns- are the national public school gains across all six NAEP subjects (4th and 8th grade Math, Reading and Science). These are across time gains rather than cohort gains, calculated by subtracting the 2009 NAEP score from the 2015 score. Looking at the blue columns shows that the national progress falls into the strictly meh: 1 point on 4th grade math, -1 point on 8th grade math, one point on 4th grade reading, 2 points on 8th grade reading, four points on 4th and 8th grade science. Nothing much to celebrate nationally.
Next look at the yellow columns- these are the 2009 to 2015 gains for Arizona school districts (no charters). As you can see, these gains are consistently larger than the blue national public numbers, especially in math and science.
Third, look at the total statewide gains (Arizona flag columns). These are the gains for the combined district and charter schools between 2009 and 2015. As you can see, these gains are consistently larger than the district gains alone (yellow columns) and far, far larger than the national public averages (blue columns). Arizona was the only state to have statistically significant gains on all six NAEP exams between 2009 and 2015.
Finally, in the back in red tower the gains for Arizona charter schools between 2009 and 2015. The over/under for percentage of Arizona students attending charters in 2015 was around 15%, so although these gains are huge, they directly move the statewide needle by the differences between the yellow district columns and the flag columns. *See boring stat nerd note below.
The above gains represent the 2015 minus the 2009 scores-for example Arizona’s 8th grade math score minus Arizona’s 2009 8th grade math scores. A different method for calculating NAEP gains is to follow the progress of a single cohort of students across time. The NAEP math and reading tests have been timed and scaled to allow such comparisons- 4th graders took for instance the 4th grade NAEP math in 2011 and the NAEP 8th grade math exam in 2015. So…which state’s students learned the most about math between their 4th grade test in 2011 and and the 8th grade test in 2015?
Nationally American students gained 41 points between the 2011 4th grade exam and the 2015 8th grade exam-so nationally about 10 points per year. Arizona lead the nation with a 48 point gain. So how did Arizona charter schools fare in this comparison? **See second nerdy statistical note.
Note that the gap between Arizona charters and districts in cohort gains (12 points) is almost as large as the gap between gain leading Arizona and the lowest rated state (Alabama). So what does this mean in practical terms? The faster rate of improvement meant that Arizona charter school students got to do this on the 2015 NAEP, which is pretty cool if you like majority-minority schooling sectors that show globally competitive levels of academic achievement:
Finally, we have a rich set of empirical studies that suggest that parental choice leads to academic gains in traditional district school systems. Going back to the first slide, we have reason to suspect that some of the differences between the yellow and the blue columns relates to parental choice. If you suspect that budget cuts lead to academic gains (I don’t) then okay maybe, or if you can come up with a reason why new standards would have a very unusually large positive impact in Arizona when they flopped around the country, I’m willing to entertain a story to that effect, but it sounds like an implausibly complicated story.
On the choice side, round about 2007, the economy collapsed in a way that made a lot of property available, and Arizona’s charter sector put the peddle to the metal. Arizona charter school sector rose by a rousing 43,000 students in 2013 for example. The number of students exercising private choice also increased during this period, and statewide enrollment growth slowed, but that increase pales next to that of Arizona charters, which increased from 95,000ish students in 2008-09 to 188,000ish students in 2013-14.
We can feel confident that some of the difference between the statewide numbers and the blue columns relates to parental choice. We can feel very confident indeed that some of the difference between the red columns and the blue columns is related to parental choice. I’m open to other interpretations-and the comment section is open-but Occam’s Razor leads me to believe that a huge increase in the prevalence of parental choice that occurred during the Great Recession lead to direct benefits (high charter school scores) and powerful competitive effects (attract students or suffer real consequences- real accountability as opposed to the phony slap on the hand sort).
*The 2009 NAEP Arizona charter school estimates had large standard errors of estimates, owing to the considerably smaller size of the sector at the time. Unless Arizona charter school sleeper agents have infiltrated the NAEP there is little reason to suspect that random error will consistently advantage charter schools across six NAEP exams. Random error in both the 2015 and 2009 estimates means that the red columns in the first chart could be either smaller or bigger if we had actually tested everyone, but I’m at a loss for a reason to think of a reason why the errors across twelve different testing samples (six in 2009 and another six in 2015) would consistently line up to produce a mirage of Arizona charter school academic conquest, again absent sleeper agent infiltration.
** Standard error plays into the calculation of cohort gains as well on both ends of the calculation (in this case 2011 and 2015 scores) such that either could have been higher or lower if we had tested the entire population. Standard errors are larger for sub-population estimates than statewide averages, but again could play either way. For example if the population score for 2011 charter school students was higher than the NAEP estimate, the cohort gain will be overestimated, and if the true population score in 2011 was actually lower, then the cohort gain reported here would be an underestimate. All NAEP scores are estimates based on samples. Arizona’s charter school students displayed larger than any state cohort gains than any other state in both math and reading, but we cannot have the same level of confidence in these estimates as in statewide averages. Again, assuming random error and a lack of Arizona charter school sleeper agents in NAEP, we would not expect random error to consistently advantage Arizona charter schools.
Finally, the state’s AZ Merit exam also shows large advantages for Arizona charter school students vis a vis district students. Sampling is not an issue in AZ Merit, and these results lend reinforcement to the NAEP results. Unless…AZ charter school sleeper agents infiltrated the state’s testing system as well…
Jay P. Greene's Blog 
I have two thoughts. You ask why AZ progressed the most from 2009 through 2014 and you look to the relative performance of Charters vs. Public schools as an answer and perhaps the only answer.
But let’s consider one other possibility. AZ’s performance in the early part of this century was so pathetic that we had a long ways to go just to approach national averages. Parents knew it. Especially parents moving to AZ from other states. We have a large population of former residents of MN, IA, IL where public schools perform at high levels. The shock of leaving MN K-5’s and putting your kid in an AZ K-5 realizing your kid learned all that stuff 1 or 2 grade levels before in MN shakes up the community fast.
So why did public schools improve sharply? First, parents got active and angry and demanded it. Second, Charters provided the competitive option that forced it.
The generation of neglect of AZ kids is ending. A mid term election in 2018 could flip the legislature creating yet one more dynamic shove forward for our K-12’s through some improvements in funding.
And the combined forces of parents demanding improvement plus Charters providing a true “choice” will continue to drive AZ education forward. Don’t expect to lead the nation in relative improvement forever. We sandbagged to win that title – and we wrecked the educations of kids from 2000 to 2014 to do that.
Mike-
There is nothing inevitable about low initial scores that guarantees gains- plenty of states started low and stayed there. I suspect that funding will increase at some point, but again decades of research indicate that there is zero guarantee that additional funding will result in improvement.
Abysmal starting points don’t guarantee success. But once progress begins, the initial gains will always stand out and seem massive by comparison to others who were “trying” all along.
As for the funding question, that too is a question that depends on your starting point. If schools are adequately funded, more funding just spends more money. When schools are deprived of critical resources, a little more funding drives a quick improvement in performance.
Let me be clear – I am not talking about more funding for District Overhead, Buses, et al. I am talking about funding precisely aimed at the classroom. More and fairly paid teachers with the right classroom resources will, in my opinion, move the AZ performance needle ahead by a big margin for our next significant incremental gain in statewide performance.
That last part’s not exactly true Ladner. There’s been an increasing body of research showing the academic benefits of increased investment in low-income schools. Here’s an example: http://www.nber.org/papers/w22011
Geoff-
I was careful not to state that more funding would not result in improved outcomes, just that there is no guarantee that improved funding would improve outcomes. There is an abundance of evidence on that front, but I don’t pretend to fully understand the relationship between funding and performance, just that we have been bitterly disappointed on that front in the past.
Mike-
Please tell me more about what you mean by other states were “trying” in the past while Arizona wasn’t. The voters passed Prop. 301 increasing funding before I got to the Cactus Patch, and Governor Napolitano increased funding on a per pupil basis. The voters also endorsed the FTF preschool program, etc. If I recall correctly, Arizona’s tax effort per $1000 of income is not super low ranked.
Still needed: raw numbers on grade 8 students, especially in charts that mix charter school test score averages with regular public school test score averages (and, then, the numbers at the varying performance levels).
Performance levels are here: https://nces.ed.gov/nationsreportcard/naepdata/
I don’t think anyone is tracking grade level charter school attendance, but why would you want to see it? As far as the NAEP goes the important thing is not the number of 8th graders but rather the number of 8th graders in the sample (and no one but NAEP knows that) but that sort of thing gets reflected in standard errors.
Good questions – did the entire State of AZ give up? No, it didn’t, and I should be more specific. Scottsdale Unified School District went in to self-destruct mode ten years ago and is just now beginning to sort out the suicidal behavior of the Dave Peterson regime.
In short order DP eliminated most of the programs for accelerated learners – AP, Gifted, Honors; systematically cut out the highest paid teachers without regard to whether they were also the best performing teachers; increased class sizes; eliminated graded homework in K-5’s; eliminated traditional grading; shifted the primary focus of the schools to avoiding non-proficient scores while ignoring customer demands to challenge the higher performing students; decimated the math programs of the District; cut Specials; and at his low point cut a half day out of each Wednesday in all the K-5’s in Scottsdale.
It is hard to imagine more destructive actions that could have been taken.
At one point I listened to him and his Senior Leadership Team tell the Budget Committee that wiping out programs was needed to “Teach Voters a Lesson about what happens when you refuse to pass an Override”.
After a decade of that abuse, SUSD test scores dropped consistently. Enrollment was lost even after busing in over 2,000 out of district kids.
Now the District is wrestling with the pain of restoring academic excellence. It will go through a 3-4 year period of sharp improvement in scores simply because once you stop trying to swim to the bottom of the ocean, salt water buoyancy will drag you back up to the surface (if you didn’t choke and fill your lungs with sea water).
So…maybe those massive wait lists for Scottsdale charters are there for multiple reasons.
You are right, Matthew.
Dave Peterson did teach parents and voters a lesson. The lesson they learned was to get on lots of Charter School wait lists.
Probably the single most valid indicator of how SUSD does Post-Peterson will be what the Wait Lists look like in 2020.
Charters will need to step up their game to continue to put up flashy numbers as they take on a bigger slice of the student pie. And SUSD seems focused on restoring competitive equilibrium rather than just whining about an unequal playing field.
MATT: I didn’t read your material word for word, just close enough to get the sense that Arizona charters were helping boost nearby public school achievement as well. That’s definitely NOT been the case for Scottsdale Unified (SUSD), within which BASIS Scottsdale resides. SUSD pupil achievement has stagnated/declined for about a year, and now per the latest USNews/World Report data is below the Arizona action. (This of course also makes SUSD even further below the National average. SUSD is also at/somewhat below the Global average.
How could that be possible? SUSD hid those adverse trends until forced into view via a FOIA action this August. Since then it has yet to set goals for anyone at any level for improving pupil achievement. (It also hasn’t even corrected the excessive transportation and overhead expenditures pointed out by the Auditor General in 2012.) It’s Board rarely discusses pupil achievement, preferring to follow the path detailed by F. Northcotte Parkinson about a century ago in England’s Department of the Navy. And its still using the same old reportedly weak Springboard curriculum today – again, despite years of complaints from leading teachers.
Anyway, even without much discussions and data, Board members suspected pupil achievement was falling – especially in math. Yet, after almost 7 years service (for one) and almost 5 for another, they did nothing but complain. It’s hard getting Board members largely elected by/dominated by teachers to do anything.
Finally, I assume you’re aware that of mysterious gains in NAEP 8th and 4th grade scores (especially the latter) over a number of years that nobody has explained that I’m aware of. I instead focus on 17-year-old scores. CATO Arizona SAT trends (1972 – 2012) basically show NO improvement, and I can’t find current or trend Arizona NAEP 17-year-olds’ data.
So, what are we left with? Damned if I know.
I’m afraid that Scottsdale Unified, or most any other district for that matter, keeps almost unlimited ability to run things down, but at least parental choice gives a strong feedback loop to encourage them to do otherwise, and other options for parents.
Matt,
1. Attendance is not enrollment. Each is an important variable but they point to different things.
2. Percentages of sampled students at 4 different performance levels (NAEP) give us a lot of information at grade 8 and grade 12.
3. The average score of the total number of NAEP-sampled grade 8 kids of about 100,000 regular public school kids at that grade level in the state (AZ has about the same pop. as MA does) means a lot more educationally speaking than the same average score based on the total number of NAEP-sampled grade 8 kids in a state’s charter schools in grade 8–which is a much, much smaller number.
While AZ has a lot of kids in charters (about 180,000), the majority are in elementary school charters (as in most if not all states). One can find grade 8 enrollment in Basis schools and in Core Knowledge schools (and in some others), but getting a very large number of kids to a particular score is much more of an achievement than getting a very small number of kids to that score. Especially when the Basis and CK schools are older charters that took in kids of parents years ago (I knew some of them in AZ) who wanted a classical ed for their kids.
One also needs to look at percentages of kids at the highest performance level in grade 8 for an understanding of how many authentic STEM students in high school we may have in the future. Grade 8 tests today mostly address arithmetic-related stuff. Comparing most kids in a state today to small numbers in charter schools in grade 8 doesn’t even begin to tell us what we need to know, especially if ed researchers think the math textbooks used and the reading level of the texts teachers assign in ELA are up to the parents to figure out. Most don’t know how to judge them.
You lost me at “attendance is not enrollment” but I think the factors you describe would be reflected in the standard error of estimate, which has declined substantially since 2009.
Matt, why do you keep telling us there’s a huge, beautiful forest here when all I can see are these crummy little trees?
https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwjWx4iXm9LTAhXFFZoKHa0HCH8QjRwIBw&url=http%3A%2F%2Fbestanimations.com%2FNature%2FNature.html&psig=AFQjCNHJ4vo6OElUc86ATkxiFf-Mmq35Aw&ust=1493848957629706