The Origin of Arizona’s Nation Leading NAEP gains

May 1, 2017

(Guest Post by Matthew Ladner)

It’s been interesting for me to watch different columnists at the Arizona Republic react differently to the news that Arizona is the only state that has been the only state with statistically significant gains on all six NAEP exams. Bob Robb stated in a recent column that he supports choice but thinks it is limited strategy for improving outcomes. His colleague Joanna Allhands noted the fact that Arizona has lead the nation in NAEP gains, but said we have no idea of why that was the case.

I hope the above chart explains why I think even Robb is selling choice short, why Allhands should reconsider K-12 agnosticism. Formally Allhands is correct that we do not know why Arizona has been leading the nation in gains, but I hope a close examination of the above chart will be fairly persuasive that Arizona’s choice policies had a great deal to do with it.

So let’s peel the above chart like an onion. The first layer- blue columns- are the national public school gains across all six NAEP subjects (4th and 8th grade Math, Reading and Science). These are across time gains rather than cohort gains, calculated by subtracting the 2009 NAEP score from the 2015 score. Looking at the blue columns shows that the national progress falls into the strictly meh: 1 point on 4th grade math, -1 point on 8th grade math, one point on 4th grade reading, 2 points on 8th grade reading, four points on 4th and 8th grade science. Nothing much to celebrate nationally.

Next look at the yellow columns- these are the 2009 to 2015 gains for Arizona school districts (no charters). As you can see, these gains are consistently larger than the blue national public numbers, especially in math and science.

Third, look at the total statewide gains (Arizona flag columns). These are the gains for the combined district and charter schools between 2009 and 2015. As you can see, these gains are consistently larger than the district gains alone (yellow columns) and far, far larger than the national public averages (blue columns). Arizona was the only state to have statistically significant gains on all six NAEP exams between 2009 and 2015.

Finally, in the back in red tower the gains for Arizona charter schools between 2009 and 2015. The over/under for percentage of Arizona students attending charters in 2015 was around 15%, so although these gains are huge, they directly move the statewide needle by the differences between the yellow district columns and the flag columns. *See boring stat nerd note below.

The above gains represent the 2015 minus the 2009 scores-for example Arizona’s 8th grade math score minus Arizona’s 2009 8th grade math scores. A different method for calculating NAEP gains is to follow the progress of a single cohort of students across time. The NAEP math and reading tests have been timed and scaled to allow such comparisons- 4th graders took for instance the 4th grade NAEP math in 2011 and the NAEP 8th grade math exam in 2015. So…which state’s students learned the most about math between their 4th grade test in 2011 and and the 8th grade test in 2015?

Nationally American students gained 41 points between the 2011 4th grade exam and the 2015 8th grade exam-so nationally about 10 points per year. Arizona lead the nation with a 48 point gain. So how did Arizona charter schools fare in this comparison? **See second nerdy statistical note.

Note that the gap between Arizona charters and districts in cohort gains (12 points) is almost as large as the gap between gain leading Arizona and the lowest rated state (Alabama). So what does this mean in practical terms? The faster rate of improvement meant that Arizona charter school students got to do this on the 2015 NAEP, which is pretty cool if you like majority-minority schooling sectors that show globally competitive levels of academic achievement:

Finally, we have a rich set of empirical studies that suggest that parental choice leads to academic gains in traditional district school systems. Going back to the first slide, we have reason to suspect that some of the differences between the yellow and the blue columns relates to parental choice. If you suspect that budget cuts lead to academic gains (I don’t) then okay maybe, or if you can come up with a reason why new standards would have a very unusually large positive impact in Arizona when they flopped around the country, I’m willing to entertain a story to that effect, but it sounds like an implausibly complicated story.

On the choice side, round about 2007, the economy collapsed in a way that made a lot of property available, and Arizona’s charter sector put the peddle to the metal. Arizona charter school sector rose by a rousing 43,000 students in 2013 for example. The number of students exercising private choice also increased during this period, and statewide enrollment growth slowed, but that increase pales next to that of Arizona charters, which increased from 95,000ish students in 2008-09 to 188,000ish students in 2013-14.

We can feel confident that some of the difference between the statewide numbers and the blue columns relates to parental choice. We can feel very confident indeed that some of the difference between the red columns and the blue columns is related to parental choice. I’m open to other interpretations-and the comment section is open-but Occam’s Razor leads me to believe that a huge increase in the prevalence of parental choice that occurred during the Great Recession lead to direct benefits (high charter school scores) and powerful competitive effects (attract students or suffer real consequences- real accountability as opposed to the phony slap on the hand sort).

*The 2009 NAEP Arizona charter school estimates had large standard errors of estimates, owing to the considerably smaller size of the sector at the time. Unless Arizona charter school sleeper agents have infiltrated the NAEP there is little reason to suspect that random error will consistently advantage charter schools across six NAEP exams. Random error in both the 2015 and 2009 estimates means that the red columns in the first chart could be either smaller or bigger if we had actually tested everyone, but I’m at a loss for a reason to think of a reason why the errors across twelve different testing samples (six in 2009 and another six in 2015) would consistently line up to produce a mirage of Arizona charter school academic conquest, again absent sleeper agent infiltration.

** Standard error plays into the calculation of cohort gains as well on both ends of the calculation (in this case 2011 and 2015 scores) such that either could have been higher or lower if we had tested the entire population. Standard errors are larger for sub-population estimates than statewide averages, but again could play either way. For example if the population score for 2011 charter school students was higher than the NAEP estimate, the cohort gain will be overestimated, and if the true population score in 2011 was actually lower, then the cohort gain reported here would be an underestimate. All NAEP scores are estimates based on samples. Arizona’s charter school students displayed larger than any state cohort gains than any other state in both math and reading, but we cannot have the same level of confidence in these estimates as in statewide averages. Again, assuming random error and a lack of Arizona charter school sleeper agents in NAEP, we would not expect random error to consistently advantage Arizona charter schools.

Finally, the state’s AZ Merit exam also shows large advantages for Arizona charter school students vis a vis district students. Sampling is not an issue in AZ Merit, and these results lend reinforcement to the NAEP results. Unless…AZ charter school sleeper agents infiltrated the state’s testing system as well…

 

 

 


It’s Too Much Winning Arizona!

April 25, 2017

(Guest Post by Matthew Ladner)

The Arizona winning just does not stop-BASIS scores five of the top 10 US News and World Report’s Top 10 high schools.

You other 49 states are cordially invited to join in the winning. We’ve yet to find any point of diminishing marginal returns here in the Cactus Patch.

 

 

 

 

 


A Society that Puts Freedom before Equality will get a high degree of both

April 20, 2017

(Guest Post by Matthew Ladner)

Want proof- here is how “9/33” charter sectors did on the 2015 NAEP 8th grade math test. First let’s look at middle and high income kids in Arizona and Colorado charters compared to the statewide averages for middle and high income kids:

Whew- would you look at that? I wonder if those AZ and CO charter school kids are getting half the funding per pupil of Massachusetts or not. Yes, right, so back on track here, those above kids are all middle and high-income, so how did low-income students far in these awful, horrible, no-good Wild West anarchist charter schools perform? I’m glad you asked:

 


Community Inclusiveness by State Charter Sector

April 12, 2017

(Guest Post by Matthew Ladner)

The National Alliance for Public Charter Schools dashboard has lots of cool data on state charter school sectors. I decided to take a look at the geography data by state to see how some of my favorites do on community inclusiveness. I don’t believe that we are likely to find charter sectors that precisely match the districts for a number of reasons, but large imbalances carry substantial drawbacks imo.

Bad:

 

Better:

Best:

Note that all three states over-represent in cities and there are practical (population density) and morally compelling (lower average performing district school options) reasons to do so. Texas however has focused on urban areas to such an extent that in communities that elect most of the state’s dominant political party (urban Texas is deep blue, with “urban” trumping “Texas” in voting preferences) schooling remains mostly a take it over leave it proposition from the districts. I heard from a CMO that operates charters in both Arizona and Texas that suburban demand for charters was even stronger than in Arizona in terms of generating wait-lists. In Arizona charter school competition is everywhere and even Scottsdale Unified needs and wants out of district transfers. Texas suburban districts meanwhile surround their new schools with trailers and keep increasing property taxes, and have limited interest in out of boundary transfers.

The “Size Does Matter” mantra again comes into play as the share of Arizona city children attending charter schools is likely considerably higher than the same share in Texas despite the fact that Arizona is more inclusive of towns, suburbs and rural areas (Texas has more urban charter schools than Arizona in absolute terms, but a much larger urban student population as well.) Thanks you again 9/33 NACSA scored charter law!


Size Does Matter Visual Aid Followup

April 5, 2017

(Guest Post by Matthew Ladner)

I posed the question a couple of weeks ago as to why the National Association of Charter School Authorizers would score the charter school law of Nevada three times higher than those of Arizona and Colorado. The Arizona and Colorado charter laws after all have a delightful record of actually producing charter school seats, and both states rocked the NAEP like:

I, ah, have yet to receive anything approaching a coherent response. So I thought I would create a little visual on seat creation. The below chart shows charter schools opening in NACSA’s second rated state (NV) and also Arizona (which had a 9/33 score for most of this period) for the period Nevada has had charter schools (Arizona got a five-year head start). Just to practice my chart making skills, I inserted the flags of each state as the chart filler.

What is that you say? No Nevada is in there– it is just a little hard to spot. See that slightly different color blue smidge at the bottom of the Arizona flag? That is the Nevada flag. Here-let me give you a better look:

I think you have a good idea of what the Arizona flag looks like. This is similar to the NAEP data explorer not being able to give us academic scores for charter school students in Nevada because, well, like the visual the sector is just too small. In fact, when you examine NACSA’s top 10 ranked charter school laws, the list correlates strongly with small sectors, weakly with impressive results.

I want to make sure that my friends in the Silver State understand that I do not want to be understood to be critical of their efforts-I want nothing but the best for Nevada charters. Rather I want to not set the bar too low on defining charter school success, nor to misunderstand what actual success in charter schooling looks like. Hint: success looks a lot like Arizona and Colorado.

The comment section as always remains open if someone would like to explain why a rational person would score Nevada’s charter law three times higher than Arizona and Colorado. And by “any rational person” I mean “any rational person who supports charter schools” as obviously lots of people who do not support charters would have ample reason to rank Nevada’s law three times higher than Arizona or Colorado’s- as from that perspective delightfully contained.

 


Quotes of the Day-Colorado Charter Schools

March 29, 2017

(Guest Post by Matthew Ladner)

The Denver Post ran a story about charter schools flourishing in Colorado. Money quote on the blessings of creative destruction:

“Take the roughly 1,700 public schools in Colorado, multiply that by 20 years, and the odds of a district-run public school being shut down by the state is 34,000 to one,” said Alex Medler, a Boulder-based consultant who worked for six years as vice president for policy at the National Association of Charter School Authorizers. “Compare that to one in 10 charters closing — one in five for Colorado — and you’ll see the imbalance.”

“The lopsidedness of district-run school versus charter public school accountability is striking,” Medler said.

Quite right- Colorado charters face real accountability conducted frontier justice style by parents with charter schools facing a one in five shot of being closed. District schools meanwhile face faux accountability (mere bureaucratic compliance) and a 1/34,000 danger of being closed.

That was the great quote in the article. The unintentionally humorous quote:

Rico Munn, a former state board member and now superintendent of Aurora Public Schools, said the board was fairly balanced in the past when it came to charter appeals. But lately, the board appears to believe charters should “rise and fall only on parental choice” and not any other factors.

Hmm, rising and falling only on parental choice. That would be consistent with the NACSA 9/33 score for Colorado on their recent rankings. I wonder how all this letting things rise and fall only on parental choice business works out academically. Oh, that’s right we actually have scores:

 


Pssst…NACSA…Size DOES matter

March 16, 2017

(Guest Post by Matthew Ladner)

I was looking at NACSA’s 2014 ratings for state charter school laws, as these would have been the most relevant before the most recent NAEP. Hailing from the out west, I noticed that Nevada had a score (26) nearly three times the score of both Arizona (9) and Colorado (9).

Longtime readers of JPGB of course will be aware that charter school students in both Arizona and Colorado rocked the 2015 NAEP exams like nobody’s business. Nevada on the other hand has had a very difficult experience with charter schooling. A decade ago or so ago I wrote a study for the Nevada Policy Research Institute that basically concluded that Nevada was missing out on high quality schools and the opportunity to relieve overcrowding in the public schools, and so should follow the example of their Arizona neighbor and get in the game. I wrote one of the earliest Jayblog posts on the subject called Fear and Loathing in Carson City:

Nevada, by comparison, has been hesitant in expanding parental options. In the five states surround Nevada (Arizona, California, Idaho, Oregon and Utah) and these states have 482, 710, 30, 81 and 60 charter schools respectively, collectively educating hundreds of thousands of students. With only 22 charter schools, Nevada is the tortoise of the region.

On November 30 of 2007, the Nevada Board of Education voted 8-0 to impose a moratorium on the approval of new charter schools. Board members told the press that the freeze was necessary because the state Education Department is being “overwhelmed” by 11 charter applications.

Arizona’s State Board for Charter Schools oversees 482 Arizona charter schools with a staff of 8. Nevada’s board overseeing cosmetology currently has 14 full-time employees.

The fear and loathing in this case referred to the sad fact that many in Carson City obviously feared and loathed charter schools. Imagine then my surprise to see a national charter school organization rank the same law a few years later as nearly three times higher than laws in nearby Western states that had produced far more opportunities for kids. Out of curiosity, I decided to check the 2015 NAEP scores for Nevada charter students.

NAEP yielded no information: the Nevada charter sector remains too small to reliably sample.

Now to give you some perspective on this, NAEP lists 5.6% of students in Nevada as Asians, and the data explorer will give you a number for male Asian students taking the 8th grade math exam in 2015 (Nevada’s Asian males did well on 8th grade math btw) but nothing for charter students of all sorts. NAEP cannot reliably sample charter students in Nevada because in 2013-14 they still only had 34 charter schools according to the National Alliance for Public Charter Schools. Those numbers for Arizona and Colorado btw were 605 and 197 respectively. Presumably the Nevada law would eventually like to have some “charter schools” result from their “charter school” law?

Next I decided to check out the 2015 rankings. The top 10 state charter sectors did not exactly cover themselves in 2015 NAEP glory. Indiana and Nevada tied for first place in their rankings, but neither state would yield NAEP estimates for performance. Alas Indiana’s 75 charter schools were not yet getting the job done in terms of scaling into the NAEP. The Nevada law passed in 1997 and Indiana in 2001. Hopefully this will get better in the future but for now:

The third ranked state (Ohio) has scores reported but those scores are consistently low, but Ohio’s NACSA score only recently increased. NACSA gets a Mulligan on that one, but continuing down the top 10 list however one fails to gain confidence regarding Ohio’s future prospects. Alabama ranks fourth, but is a relative newcomer to charter schooling so also cannot report scores. The Texas charter law ranks fifth on the NACSA 2015 list but has charter scores on NAEP that have yet to impress. I am a Texan who became an Arizonan and I would not swap charter laws with Texas even if they threw a shale formation into the bargain. In the vast majority of Texas schooling is still “take it or leave it” from the districts, whereas in Arizona even our Beverly Hills type districts are anxious for you to open enroll from outside the boundaries.

The same applies to seventh ranked Minnesota- the nation’s oldest charter law. You can get NAEP scores in MN, and I can’t thank them enough for inventing charter schools, but the feeling I get from MN charters is that they are safely contained rather than dynamic. The last three states in NACSA’s top 10-Mississippi, Missouri and South Carolina- all lack enough charter students to meet the minimum NAEP reporting requirements as well. Louisiana comes in 10th.

If you are scoring at home, NACSA’s top 10 is composed of six states with charter sectors that can charitably be described as “wee-tiny” and three others that have yet to flourish like an Arizona or Colorado, and then Louisiana. Tenth rated Louisiana’s charter sector does well in the NAEP, so bully for them, but they obviously have a unique charter history. Notably absent from NACSA’s top 10- very healthy charter sectors like Florida and Washington D.C.

Not to jump to any premature conclusions, but it appears that NACSA’s rating may be overly concerned with bureaucratic compliance rather than performance- either of the academic sort, or the “actually produces charter schools” kind. Arizona and Colorado produced hundreds of charter schools with NAEP scores that compare favorably to New Hampshire (and sometimes Massachusetts) with a 9 score from NACSA. I for one would like to see what they could do with, say, a five score from NACSA. What’s that you say? Three? Ok fine let’s try it out if you insist!

Now maybe I am missing something here, and that is why the comment section is open. I’ll leave you with the following question to consider- Nevada public schools suffer from catastrophic overcrowding. Public education in Las Vegas for a great many students involves sitting in a portable trailer being taught by yet another long-term substitute teacher. Clark County starts each school year with thousands of open teaching spots they are desperate to fill, and their officials told the New York Times they could build 23 new elementary campuses and they would be overcrowded on day one. The United States Census Bureau sees no end in sight for enrollment growth.

Please tell me why any Nevadan in their right mind would prefer Nevada’s charter law to what we see in Arizona and Colorado. I mean maybe scale and great results is not everyone’s cup of tea, but any port in a storm right?