So out riding my bike on the canal this morning I had the idea for a new visualization for cohort NAEP gains. Here’s what it looks like:
So a bit of explanation: the shotgun blast at the lower left part of the chart are 4th grade math and reading scores for states in 2013, with Arizona charter school students included. Arizona charter school students didn’t blow anyone away with their math and reading scores as 4th graders in 2013, but this is in the range of what many would expect from a majority minority school system operating with very modest funding.
Fast forward the clock to 2017 and those kids were 8th graders, which are the shot gun blast of dots on the upper right. Lo and behold, that majority minority student population is **ahem** outscoring states that spend more than twice as much per pupil and have the advantaged end of the achievement gap stick. Arizona charter students pulled this off despite spotting such states a head start in the form of higher 4th grade scores.
Wait…I’m picking up a disturbance in the Force. I can feel you thinking “Ok but students come and go from charter schools and this must explain some of those gains.”
Actually kids do come and go from charters, but to the extent this is happening Arizona charters are sending out kids with higher levels of academic achievement and bringing in kids with lower levels of achievement. From the Center for Student Achievement:
So if numeracy and literacy are an important part of what you are looking for in a school for your child, you might want to move to Arizona. Once here you can consider enrolling your child in one of our pluralistic charter school offerings which focus on everything from the arts to equestrianism to the classics. As far as I can tell, it’s the finest system of public education in the country, and it is available to you free of charge delightfully without a crushing level of taxation. Plus…bring your golf clubs:
The full implications of this new analysis from the Arizona Charter School Association showing that Arizona charter schools receive district transfers who are below statewide averages on AZMerit, and send students to districts who are on average above the statewide average, can only be fully appreciated in the context of additional data.
Quick play by play on the above chart: 85,000 students transferred between public schools between the 2014-15 school year and the 2015-16 school year. The AZMerit tests were given in the spring of 2015, before the students transferred to a new school in the Fall of 2015 for the 2015-16 school year. The sending sector in other words owns the score.
A huge part of the ongoing Two-Minute Hate against Arizona charter schools has been the notion that they are engaging in systematic creaming. The problem with this story is that the above chart reveals it to not only be false in aggregate, it also reveals it to be the exact opposite of the truth. On net the districts are sending out below average students to charters, and receiving above average students in return. The districts, in short, are guilty of precisely the charge hurled (without evidence) against charters. This is not to say that there aren’t individual schools, district and charter, doing bad things, but the net of everything everyone is doing appears to be district not charter creaming.
The part about charters sending above average performing students to districts deserves a special mention. A part of the litany against charters involves an obsession over high-school attrition rates in BASIS. This has always been off base, as only a small percentage of Arizona charter students attend BASIS, and BASIS is basically the Green Berets of academics. The Green Berets have an attrition rate as well, but the people who complete the training are deadly military professionals. Arizona students are very active (85,000 total transfers in a single year in a ~1.2m student system) and do so for a large variety of reasons- social, athletic, academic etc. I’ve always thought that the students who don’t complete BASIS were likely to have been better off for the experience, and lo and behold that seems to be the case statewide across charters. In other words, when the 8th grade BASIS student chooses to attend a comprehensive school to play football as a 9th grader and brings above average academics with him, the proper response from the district should be one of gratitude rather than condemnation. When the statewide scores show districts to be sending away low-performers in droves (to both charters and other districts btw) the complaint positively reeks of hypocrisy.
But I digress…
Another part of the two-minute hate litany would have us believe that a child with disabilities has never crossed the threshold of an Arizona charter school. If however one goes to the state’s AZMerit data file, you find the statewide percentage of district children with disabilities stood at 11.1% and at 9.73% in charters on statewide ELA exams. The percentages are similar in the math exams. In a similar fashion there is a difference in rates of limited English proficiency, but nothing like what the blaring telescreen would have us believe: 6.3% for districts and 4.4% for charters.
What about the part when the double-plus good duckspeaker screams through the telescreen to tell us that charters are bastions of White segregation? Try again: 55% of charter students are non-Anglos compared to 63% of Arizona district students. There is a difference, but both sectors are majority-minority, and neither looks like either Vermont (or North Scottsdale).
Some of the difference between charter and district performance is certainly explained by differences in student demographics but here is the next shoe to drop in the AZMerit data: every single subgroup available scores higher in charters than they do in districts. Native Americans, Latinos, African-Americans, Asians, Whites, ELL, SPED, FRL etc. students all score higher in AZ charters than in AZ districts. Some of the difference is certainly owing to demographic differences, but nothing close to all of it.
So turning our attention to the above chart and pairing it with NAEP, it makes the NAEP data seem even more impressive when you consider the fact that Arizona districts are sending below average performers to charters, and charters are sending out above average performers. Despite that, NAEP shows us things like this:
So for those scoring at home, Arizona charter schools educate a majority-minority student body, receive only about $8500 per pupil in public funding, receive low-performing students on average from districts, and send higher than average performing students to districts and…scored higher than Massachusetts on the 2017 8th grade NAEP exam plus demonstrated the best 2009-2017 improvement in the country.
I fully expect our friends in the Arizona charter school skeptic community to doubt the AZMerit analysis. My recommendation to them is to file an open-records request with the Arizona Department of Education for the same data file. Crucial findings such as this deserve scrutiny and replication. The evidence currently available leads to only one conclusion: Arizona charters are working extremely for the students fortunate enough to attend them.
So kids transferring from Arizona districts to Arizona charters have below average AZMerit scores. Those transferring from charters to districts have above average AZMerit scores. This is to put things mildly not consistent with the litany against charters that claims that Arizona charters are engaging in massive creaming of students. If anything, this data suggests the opposite: that if there is creaming occurring, it seems to be in the direction of districts creaming charters.
Mind you the creaming thesis already had huge problems, the first of which being math. When you see scores improving in both district and charter schools, it makes it very difficult indeed to argue that charters have circumvented lottery requirements (which districts don’t have) to only get the high flying academic kids. If this were the case it would have been a really neat trick to see Arizona districts showing better gains than the national average despite some of the nation’s largest Great Recession funding cuts, despite a continuing trend into deeper majority-minority states and despite losing tens of thousands of their top performing students.
The average transfer from district to charter having below average scores in fact might in part explain the district gains we see, but then you get back to those charter gains…looks like they are having a lot of success getting kids who were off track back on it.
I would be very interested to see the average grade level by transfer group in this data. My guess would be the average district to charter transfer student is an elementary student, and that the average charter to district student is more of a middle school student. In any case, when you are taking in lower than average kids and sending back higher than average performing kids as a sector, this myth is…
So let me start by saying I’m not sure how other states handle charter school facility procurement laws, and I am curious about it. It is however worth noting that Arizona provides the best access to charter schools by zip code, per this Hamilton Project map from Brookings:
Hmm, maybe other states should be doing what Arizona is doing. Also worth noting is the fact that Arizona charters are not only more proximate than other states, they are also crushing the ball academically (statewide averages in blue, statewide charter averages where available in red):
Arizona just might be doing something right in this space, so my elephant says “Careful about ‘fixing’ something that is not broken.”
Here is the missing context from this article- construction firms make profit from both the construction of charter and district schools. In recent years districts in the state of Arizona have been spending at a half-billion per year annual clip on facilities despite having relatively flat enrollment growth in aggregate.
Let’s put it on the table from the outset that this number could never be zero (air-conditioners die, roofs leak etc.) Moreover, some of this spending involves the districts who are big winners in open enrollment making seats to meet demand. I can’t make any complaint about this portion of the building spree-I like it.
Having said that, $500,000,000 per year is a lot of money (enough to pay every teacher in the state $10,000 per year more) and there are hundreds of thousands of empty seats in Arizona districts, which were badly overbuilt during the boom. The Arizona School Facilities Board lists 1.4m square feet of vacant district space (approximately 35% of the total) and strangely enough almost none of this space is suitable for a charter school (in the estimation of district officials).
The word on the street here in PHX is that a handful of the big construction firms find that $500,000,000 profitable enough to invest in bond and override campaigns. So…we continue building space, including in districts with declining enrollment like Scottsdale. Maybe a reporter should look into that…
Now back to the current Harris piece. It is lacking in context, giving no information about the relative profit margins in charter versus district construction. There are no non-profits building schools in Arizona to my knowledge. Moreover, the profits in this case only come about because of the demand for the school model. Without demand, the CMO in question would lack the funds to buy the buildings-ergo no profit. Contrast this with a district system with 1.4 million empty seats but continuing to build more of them despite flat aggregate demand and at an enormous annual cost despite huge spare capacity and the costs for vacant buildings drawing money out of the classroom.
So all in all, where does charter school construction profits rank in a list of school facility issues in your opinion?
So the data went live at midnight EST. The national data is basically flat on all four tests. Most single cycle differences are not statistically significant, so I’m looking at longer periods of time, but the 2015 to 2017 champ looks to be Florida. More details on this later but I’m resisting the temptation to bring Sally back pending further analysis…but only barely!
The short run (2015 to 2017) state level data doesn’t look terribly exciting. I’m still digesting the longer run state level data, but up top there’s a chart to tide you over. NCLB required all states to take math and reading NAEP starting in 2003, so here is the full period (2003 to 2017) gains for 8th grade math and reading by state. These are numerical gains subtracting 2003 from 2017 scores by state.
The 2003 to 2017 period is a natural one to examine, but so too is the post 2009 period, for a few different reasons. The 2009 NAEP happened both in the early days of the Obama administration and during the outset of the Great Recession. NAEP redid the 4th and 8th grade science exams that year. The inclusion of “non-tested” subjects is healthy in my view given concerns over curriculum narrowing. According to my bleary eyes two states have made statistically significant gains in all six exams for the entire available period- Arizona and Mississippi. For the 2009 to 2015 period it had been AZ alone. Arizona and Mississippi were also the only states to make statistically significant gains on all of the math and reading tests during the 2009 to 2017 period. Several states however had statistically significant gains in 3 out of 4 of the math and reading exams between 2009 and 2017-including California, Hawaii, Indiana and Wyoming. On the downside, I have not yet added up the number of significant statewide declines in scores during this period, but there are many of them.
Arizona had three flat aggregate statewide scores and a decline in 4th grade math between 2015 and 2017. At first blush Arizona and Colorado charter schools crushed the academic ball again in 2017, will dig further into details/subgroups but for now:
There is more good news in statewide charter sectors, haven’t touched the TUDA yet, stay tuned.
Just as the country benefits from political diversity, we also benefit from a diversity of policy approaches at the state level. There are those who seek greater uniformity among state charter-school policies—urging that all charters should be for five years and that default closure provisions should be spelled out, among other guidelines. Such advocates should consider the success of these western states, which have chosen not to adopt such policies. The 50 states will become less useful as laboratories of reform if we adopt a single set of policies everywhere.
Many states—including three of the four featured here—have experienced high rates of overall K–12 enrollment growth, which raises the opportunity cost of imposing a stringent charter-authorizing process. It does not follow that every state should rush to amend its charter policies to match those of Arizona, Colorado, New Mexico, or Utah, but the obvious flourishing of the charter sectors there offers food for thought. Questions to consider and debate include: What factors have led to success in these states? What steps can policymakers and philanthropists take to enable parents to take the leading role in closing undesired schools? How important a role does open enrollment in suburban districts play in creating a successful bottom-up accountability system?
We don’t know the answers to these questions. But we do know that relatively freewheeling charter-school systems have prospered in multiple states. Surely we have as much to learn from these success stories as we do from the cautionary tales from states that have experienced difficulties.
Bob Bowdon interviews David Osborne and around the 8 minute mark the discussion raises the topic as to whether parents can lead the way in school closure. Osborne claims that parents cling to schools for non-academic reasons and should not be trusted with this task. Bowden raises Arizona as a counter-example, and Osborne cites research from five years ago that found Arizona charter schools had less than stellar results.
Osborne is correct that research from five years ago found less than stellar results. What one must appreciate however is that in a sector as dynamic as Arizona charters those results are ancient history. Charter schools are constantly opening and closing in Arizona. The studies referenced by Osborne have data that ended in 2012. In 2013 Arizona educators opened 87 new charter schools and closed 18 other schools. This alone was enough to mean that the Arizona charter school sector of 2013 was a different animal than the 2012 sector, but it was hardly the only change to happen that year. In addition to schools, teachers, students and administrators moved in and out of Arizona charter sector. Younger schools gained a year of experience, moving out of their shake down cruises. A professional football team can win the Superbowl in one year and fail to make the playoffs the next year, and there is a greater degree of year to year continuity in sports than in Arizona charters. Fortunately all the available indicators (NAEP and AZMerit) show over time improvement in Arizona charters.
And then, it all happened again in 2014…and 2015…and 2016…and right now. Any look into academic results in a constantly changing Arizona charter sector is merely a snapshot. This makes it entirely possible for analysts like CREDO and Marty West to have found meh results in 2012 but for NAEP to show this in 2015:
This river of course runs both ways- the 2015 NAEP is just a snapshot as well. Fortunately Arizona’s charter results were also strong in the 2015 AZMerit, got better in the 2016 AZMerit and better still in the 2017 AZMerit.
Although Arizona law requires admission lotteries and Arizona charters educate a majority minority student body, there is obviously room for multiple factors to explain the above chart. Nevertheless, sometimes a cigar is just a cigar. Arizona’s AZMerit exam also shows large advantages for charter school students. In addition, school data of the sort CREDO and West relied upon five years ago is quite messy, especially when it comes to free and reduced lunch status. I won’t go into details but let’s just say that the record keeping at the Arizona Department of Education doesn’t always cover itself in glory and in addition many Arizona charter schools do not participate in the federal program at all. The use of Free and Reduced Lunch as an independent variable has grown increasingly questionable over time, but has been problematic for a long time in Arizona.
So in conclusion, Arizona charters are crushing the academic ball without the benevolent guidance of heavy handed technocrats. The same is true is several other Western states that show either high scores, high over time progress or else both of these things:
If Mr. Osborne would care to explain why westerners should abandon what they are doing to emulate Louisiana, I’m all ears but data from five years ago is of little more than historical interest to discussions of prospective policy.
That’s right Ralphie, in an examination of Maricopa County (greater PHX) enrollment patterns it turns out that district open enrollment involves approximately twice as many students as charter schools. Which in turn means that it dwarfs private school choice-as in private school choice and charters combined will still be far smaller than open enrollment. If however we were to include private and home-schooling in this analysis, it would mean that fewer than 50% of students attend their assigned district school.
One can only describe the implications of this as far-reaching. Let’s start with the angst over private choice.
So in the districts examined by the Center, open enrollment dwarfs charter school enrollment, and statewide charter school enrollment dwarfs private choice enrollment. The conclusion inevitably follows that much of the hand-wringing and gnashing of teeth over charters and private choice in the state has been misplaced. If one views families exercising choice as “draining money from public schools” it is primarily other district schools that have been doing most of the draining!
Curses! They’ve discovered that I am the real vampire!
Now of course it is deeply misguided to view students attending a non-zoned school as “draining money” in the first place. It is the child’s money for families to direct, not the entitlement of a particular school. Just in the districts examined there are more than 10 times the number of students using open enrollment as there are ESA students statewide. Let that sink in for a moment. To the extent that choice is causing budget problems in district schools (it’s worth it, but we’ll get to that below) let the record show that it is result of district competition far more than private choice.
Next- why do we see stuff like this?
Arizona is an unlikely jurisdiction to have been leading the nation in NAEP gains since 2009. We’re not a wealthy state (yet), we have a small working age population, our citizenry has not exhibited a preference for high taxes over the last 30 years. Just last week a sales tax increase for preschool got voted down by the voters in the “People’s Republic of Tucson” almost two to one. Five years ago voters rejected a statewide measure by nearly the same margin. More recently the voters barely agreed to increase school funding by $3.5 billion without a tax increase in order to settle a lawsuit and avert a possible constitutional crisis. We are a border state going through a border state transition in student demographics. Our students, educators and policymakers didn’t read that memo with the subject line about we were supposed to lose. Instead we lead the nation in gains.
You’re either willing to take the steps necessary to drive gains or you can dream about kumbaya policies that don’t offend many. Arizona has thus far been willing to do what it takes. The very widespread process of families matching their kids to better than the zoned option fits is obviously driving this process. To think otherwise would require you to explain how Arizona has been leading the nation in gains despite all this choice business. And then there is the data:
Next let’s talk about facility funding. A murmur that has been coming out from the district folks in recent years has been “we’re double funding capital.” In other words, taxpayers paid for buildings that are sitting empty while charter schools keep building new space. Rumors of a fresh round of litigation against the state regarding district facility funding have been circulating for years. How much sense does it make to build Taj Mahal big box schools with a student population that is this mobile? Moreover much of the empty district space issue came from the state overbuilding facilities during the boom for schools that Arizona parents did not desire and desired district schools were the destination for a large percentage of these students.
In other words, this isn’t being done to districts in so much as by districts.
If we extrapolate this Maricopa data to Tucson, then Tucson Unified’s large number of empty school buildings are empty because of open enrollment more than charters or private choice, but ultimately they are empty because families chose to empty them. Yes it is an absurd waste to have people talking about converting schools into urban gardening centers in a relatively cash strapped state that has the second fastest growing student population. Yes we should find a way for someone who will run a school that parents desire to make use of those facilities.
In the end however those empty school buildings are accountability. Real accountability, not the watered down wrist-slap version of it. Choice is the only form of accountability that is decentralized and can get delightfully out of the control of central planners and into the control of families. In Arizona it has been driving academic improvement and, alas, this process causes angst. Far from shying away from responsibility, the district schools attracting the confidence of families should be proud to be driving choice and participating in the process of creating good fits and academic gains.
UPDATE: NPR interviews Ildi Laczko-Kerr of the Arizona Charter School Association and Center for Student Achievement on the data.
The chart below plots the 2015 8th NAEP grade math scores against the 2011 to 2015 NAEP math cohort gains. The below charts include state averages and the numbers for state charter school sectors in Arizona, Colorado, Idaho, New Mexico and Utah. In the NACSA ratings of state charter school laws most proximate to these data, I recall all of these states with the exception of New Mexico received single digit scores out of a possible 30 something points, clustering towards the bottom of the rankings. New Mexico still ranked pretty low. Generally these states lost points for not having default closure and similar type provisions. How did the charter schools in these state manage?
The high performance/low NACSA phenomenon looks to be a western trend. These charts are not stone tablets handed down from the mountain, but I can’t think of any reason they would systematically favor charter sectors. Those “Wild West” charter sectors look, ah, really good at math. If you recall the international comparisons, Massachusetts ranks up there near the best European and Asian countries. Let’s take a look at the reading results:
Well there you have it- AZ, CO, ID and CO all have Massachusetts like results, and it appears that when it comes to spurring reading gains, New Mexico charters are the ultimate power in the universe…I suggest Enchantment State parents use it.
Please do me a favor and email this post to the next five people you hear use the term “Wild West” as a term of derision in an education conversation. Bless their little hearts, but they generally have not bothered to look at empirical data in order to see whether it can be squared with their regional/ideological prejudices.
NAEP Reading Scores from 2015 along the horizontal axis, NAEP reading cohort gains (2015 8th grade scores minus 2011 4th grade scores). Ok so stare closely at the chart around the 262 score from the bottom to the top. Arizona, Tennessee, Oklahoma, Michigan, Rhode Island, New York, Florida and Delaware all had approximately the same 8th grade math score, but took different paths to get there. Some, like Delaware, Florida and Maryland started above the national average in their 2011 4th grade scores, but had small gains. Others like Arizona and Oklahoma, started below the national average in their 4th grade scores but grinded their way to large gains to catch up.
Keep staring at that middle portion of the chart. Is Tennessee supposed to be neck and neck with Rhode Island? Rhode Island’s 7 point lead in the 2011 4th grade reading scores and almost $7,000 per student spending gap would say no, but the Tennessee kids didn’t get the memo and ended in a dead heat by 8th grade.
Ok so spot NY on the above chart and then look at math:
Arizona, Connecticut, Kansas and Maryland had 2011 4th grade math scores of 235, 242, 246 and 247 respectively. These had current (not total) expenditures that year of $7,782,$16,224, $9,802 and $13,946 per pupil. As an Arizonan, I’m delighted to have closed the gap with Connecticut, Kansas and Maryland. If I were a taxpayer or educator in Connecticut, Kansas and/or Maryland I would not be pleased.
Now locate New York on the math chart. I guess $19,965 per pupil just doesn’t buy what it used to in New York.
Ultimately it is good news that we have examples of states with diverse student bodies making academic progress. Remember- winter is coming to state budgets as 10,000 boomers per day reach the age of 65 and health care costs continue to rise. I hope you can get that sorted out New York but in the meantime both your students and taxpayers are getting horribly short-changed by your K-12 rent-seeking groups. The founders included a solution for you in our constitutional system: federalism. Did I mention that in addition to lower taxes, it is very pleasant here in the winter? As Ling Ving once sang “New York’s alright-if you want to freeze to death!”
Be sure to bring your golf clubs:
As far as where you’ll send your kids to school, Arizona has outstanding options in the public school system in both districts and charters. Here’s some dots to connect on the average performance of Arizona charters:
Additionally if you happen to prefer a private school for your child, Arizona’s policies support your families capacity to make that decision. Tired of having the daylights taxed out of you to pay for a public school system you don’t want to put your kids in, and then paying private school tuition on top of that? I thought you might. Head south until you reach Interstate 10 and then go west young family!
Posted by matthewladner 
Jay P. Greene's Blog 