The deeper you look into the NAEP the better things look for my home state. Earlier I showed you that Arizona led the nation in overall academic gains in both 2013 and 2015. Today let’s take a close look at math, and student subgroups- both in terms of gains and overall performance.
Starting with Anglo students- how did their gains look in 2015? The chart below subtracts the 2011 NAEP 4th grade math scale points from the same cohort’s 2015 8th grade math scores for Anglos.
Ok so they led the nation in gains- but that might not matter because their scores could still be low. Except they aren’t:
Arizona is in striking distance of everyone but the absurdly gentrified right side of the tracks sections of DC. So moving on to non-Anglo students, the NAEP can also track Black students- here were gains by state in 2015.
Well would you look at that? Number one again. Hmmm. Well but it still might not mean much because the overall scores….could..still.be…….low?
Hispanic students gains/scores are merely spectacular rather than absolute tip of the spear:
Ah well it gives us worlds yet to conquer, as do the scores:
Ok so someone try to make a case that a state other than AZ deserves to be considered the pound for pound mathematics champion of the United States. I’ll have my gloves laced up and will be ready to give you a canvass nap in the comments section (unless you have a decent case to make) but I’m declaring Arizona to be the pound for pound state math champion.
For you incurable skeptics, here are a few charts on Arizona’s glorious six-year (and counting?) reign as the NAEP gain champion, starting with 2009 to 2013:
So just to provide some play by play here- Arizona 4th graders were 9 points below the national average on 4th grade math in 2009, but when this cohort reached 8th grade in 2013, they were only 4 points below the national average- within striking distance of the national average.
Here is what happened in the next cohort we can track in NAEP math- 4th graders in 2011 to 8th graders in 2015.
So in 2011, Arizona 4th graders scored five points below the national average on 4th grade math, still ranked in the bottom 10 of states. In 2015 the same cohort of students scored 2 points above the national average on 8th grade math. This was the first time in the history of the NAEP that Arizona had scored above the national average in any NAEP exam in any subject.
So where does this leave Arizona? Unfinished to be sure, but headed in the right direction. NAEP 8th grade scores are more reflective of the overall quality of a school system than 4th grade scores in my opinion, as you have additional years of schooling. In 2015, Arizona had moved to within the margin of error of the national average on both NAEP math and reading (slightly above in Math, even more slightly below in Reading). Arizona students were below the national average and outside the margin of error on both Math and Reading in 2009.
Of course the national average itself is a milestone but not a resting place- the United States does not rank well against other countries. We need to keep it up. Arizona was never going to become even an average performing state without above average gains, and no other state matches the gains seen by Arizona students over the last six years- so a bit of celebration is in order:
I’ve already been happy to report that measuring NAEP gains between 4th grade scores in 2011 and 8th grade scores in 2015, Arizona banked the largest overall gain in achievement. Just out of curiosity I decided to rank the states during the previous period (2009 to 2013) the same way- 4th grade scores in 2009 compared to 8th grade scores in 2013. I did this separately for Math and Reading.
Any guesses on which state came in first in overall gains? Here is a few clues:
Arizona wins again! First in overall math gains, fifth in overall reading gains, highest overall combined ranking. Feel free to pop out and take notes- we’ve been doing this since (at least) 2009. Yes, feel free to bring your golf clubs.
MM: Hows it goin eh? So I am Matthew MacKenzie, and this is my sister Lisa Graham Mackenzie…put on your touk!
LGM: <putting on hat> COO LOO COO COO COO COO COO COOOOOOOO! Good day!
MM: Good day! Sos our topic today is education…
LGM: Education? Do you need me to teach you how to open a beer again hoser?
MM: No! Ifs I didn’t learn that after the sixth lesson or so, I’da thirsted to death already!
LGM: You….learn? <snorts>
MM: Yeah….okay…..so good day, we are here today to talk about education in the United States.
LGM: They have education in the United States? I thoughts the test scores were even lower down there than yours hose-head?!?
MM: Yeah, well, they are, check out how high Canada is on this ranking eh!
MM: You may have to squint but Canada is near the top! United States, er, well but at least some American states are making progress…like getting a lot better…
LGM: Oh you mean like BEER! Remember when just a small number of breweries made almost all the beer, and it mostly tasted like stagnant pond water?
MM: Yeah- kinda like that! So’s now some states have kinda done the microbrewery thing for schools eh? Sos more people can find a school that they like!
LGM: Beauty! And hosers like you can find a beer you like!
MM: Yeah so the state with the biggest gains did microbrewery education in a big way eh? Ands you can read about in our column in the Arizona Republic!
The Arizona Chamber of Commerce and Industry held an awards breakfast last Friday, where they recognized a number of worthy recipients including friends serving in the legislature like Senator Debbie Lesko and Senator Steve Pierce and Representative Paul Boyer. They also however chose to recognize a rather dubious character with whom you will be all too familiar by bestowing upon him the Milton Friedman Award, Obviously I was deeply touched to receive an award named after one of my heroes.
At the request of the Chamber, I prepared the following remarks:
I am deeply touched to receive this honor, but I must say that I feel a bit like Jack Ryan. You may recall the scene in the Hunt for Red October when Ryan exclaims “Me?!? I’m just an analyst!”
Arizona is sailing into history!
While I am deeply appreciative of the award, it is I should honor you. The groundwork for what I am about to describe was already in place when I arrived in Arizona in 2003. You as long-time business and civic leaders in Arizona should take great pride in what I will relate.
It was recently reported that Arizona ranks number two in job growth. I am happy to relate to you that Arizona ranks number one in K-12 academic gains. The National Assessment of Educational Progress gives academic exams to 4th and 8th graders in all 50 states every two years. When you follow the academic progress of 4th graders in 2011 to when they became 8th graders in 2015, you find that Arizona students made more progress than any other state. Given everything this state endured during the Great Recession, this is a remarkable tribute to the resiliency of our students, educators and policymakers.
This progress is across the board and includes both district and charter schools. In addition our charter school students did something truly extraordinary. On the same 2015 National Assessment of Educational Progress, Arizona’s charter school students scored comparably to the highest performing states like Massachusetts and New Hampshire. This is all the more impressive when you consider that Arizona charter schools are funded far more modestly, and have student bodies far more diverse, than the schools in New England.
These results are remarkable. How did this happen? What is the secret sauce? There is no single explanation and there are many ingredients in the Arizona K-12 reform gumbo. You made the mistake however of giving me an award named after the great Milton Friedman and then the even larger mistake of giving me the microphone, so I am going to talk about parental choice. It seems clear to me that parental choice has been a major contributor to Arizona’s improvement.
Parental choice is controversial. Some people believe that parental choice is about some schools being “good” while others are “bad.” Those who believe this however are mistaken. Parental choice is about the fact that every single child deserves to have access to a school that is a good fit for them. Good fits between students and schools are very powerful, and we cannot replace it with any other source of improvement. Without giving parents the ability to match the needs and interests of their child with the particular strengths of a school, the public education system will never reach full potential.
During the campaign, Governor Ducey quite rightly placed an emphasis on Arizona students sitting on wait lists at our high demand district and charter schools of choice. These students only have one shot at their K-12 education, but they find themselves stranded by the inadequacies of our policies, waiting for the opportunity to attend their good fit school. Meanwhile the sand continues to run through their hourglass.
Our challenge includes this, but it is also more than this.
Tens of thousands of Arizona students sit on wait lists, but hundreds of thousands of Arizona parents never even considered some of our highest performing district and/or private schools. These schools may have been great fits for the needs of their children, but they didn’t even cross the radar screens of these parents for consideration. Why not? Because they have effectively been priced out of consideration. Parents either cannot afford the high price of real estate for the district schools, or else cannot afford to pay tuition in addition to their taxes. Many sadly see these schools as being for someone else, but not for them. It doesn’t however have to remain this way. We have it in our power to make our educational opportunities more inclusive. The blessings of liberty should not remain the privilege of the few, but rather the birthright of all.
I fell in love with Arizona because of our innovative spirit and I believe that we have been richly rewarded for it. If Dr. Friedman were still with us, I believe he would be proud of what we have done, and would encourage us to do more. Arizona is a state with big horizons, where the sky is the limit. May we always remain so.
I genuinely am deeply appreciative of both the award, and the opportunity to work with great people on these issues in Arizona.
NAEP has been following an odd number year cycle, meaning that the four-year gap between the same cohort of student can be measured. In this case, I took the statewide 2015 Math scores and subtracted from them the statewide 2011 4th grade math scores. We more commonly compare 8th grade scores to those of previous 8th graders, but measuring cohort gains is also of interest.
A note of caution before proceeding it is possible to do well on the below gains analysis without ranking terribly high. Likewise you could have the highest overall 8th grade scores but also had high 4th grade scores four years earlier and look meh in this particular analysis. Like achievement gaps, each state’s gain score requires a close look before drawing conclusions- but generally speaking the states with the biggest scores below would have started with modest 4th grade scores and then shown much higher scores for the same students as 8th graders. Obviously factors other than school system effectiveness could come into play (massive gentrification in the District of Columbia comes readily to mind) and every state will have students both come and go between 2011 and 2015.
Okay now that you’ve read the warning label:
In terms of context, Arizona’s 4th graders scored 5 points below the national average in 4th grade math in 2011 but two points higher in 8th grade math in 2015. There is nothing about starting below the national average that makes it inevitable you will crush the ball in gains. Alabama for instance had 2011 4th grade math scores 9 points below the national average but then displayed 8th grade scores in 2015 that were 14 points below the national average.
Reading gains below:
Arizona’s 4th graders scored 8 points below the national average in 4th grade reading in 2011. As 8th graders in 2015 however this group of students had scores only one point below the national average- within the margin of error thus catching them up to the national average. In short, Tom Patterson nailed it- AZ looks to have had the best 2011 to 2015 period.
So this got me to thinking-what if we tracked the same cohort gains for Arizona charter school students and compared them to statewide averages? As we’ve noted before, with the largest state charter sector as a percentage of the population, Arizona has substantially more students than the entire public school system of Wyoming. The National Alliance of Public Charter Schools also compiles data about state charter sectors, so we know that the sector (like Arizona as a whole) has a majority minority student body.
Well this is what it looks like for Math:
Keep on rocking the free world! Similar story with reading:
Given the Arizona’s charter schools rank near the top in over NAEP scores, and first in overall NAEP cohort gains between 2015 and 2011, they have a great deal to celebrate- as does Arizona as a whole. Nothing in these results makes a case for complacency Arizona has only numerically exceeded one of the national averages on the four main NAEP exams, and America remains a low performing nation. Nevertheless if this is what getting an F on the Ravitch report card looks like, what can we do to get an F minus minus?
New York and North Dakota are your team captains of the Washington Generals in the spending up NAEP scores down sweepstakes. Meanwhile out west AZ and California did more with less.
In our next exciting episode of reviewing the DC education scene from the new study I wrote with Heritage we take a look at where the District remains despite a stunning level of investment and 25 years of improvement.
So let’s put DC into context in terms of revenue per pupil with data from the United States Census Bureau.
DC charter schools seem to be bringing in somewhere in the neighborhood of half of that figure on a per pupil basis. Now let’s see what the Trial Urban District Assessment NAEP has to say about how DC kids compare. The chart below compares kids on NAEP 8th grade math scores for students whose parents graduated from high-school but did not attend college. The hope here is to rank districts by kids not born on third base.
Note that if you do the same comparison by FRL status instead of parental education you still find DC ranked only ahead of Detroit and behind everyone else. The 17 point advantage for DC charter school kids in the above chart is considerable, but as the comparison makes painfully obvious, DC charters may be on their way, but they have not arrived. Still with less money and better scores the ROI is far, far higher than DCPS.
The heartbreaking part of the story however lies with the DCPS students. I’m not going to bother to look up the revenue per pupil statistics for Detroit but I am putting the over/under at half that of DC. Decades into DC reform efforts DCPS remains largely unchanged- far better at spending money than at teaching children, other than those who bought or worked their way into the high performing schools.
Next episode we’ll discuss what to do next. It shouldn’t involve continuing to bang our heads against the “better scores through improved management in an utterly broken system” wall. It also does not involve giving up.
Recently I made an off-hand comment about Arizona NAEP gains being like the problem with the orbit of Mercury. I decided that it would profit from some further explanation. Newtonian mechanics seemed to have everything figured out, with that nagging problem of the orbit of Mercury doing something it shouldn’t. The “problem” with the orbit of Mercury of course wasn’t really a problem at all. It turned out to be a problem with our incomplete understanding of how the universe works- as illustrated in the above video.
So just how do Arizona NAEP scores resemble the orbit of Mercury? The 2015 NAEP shows that Arizona charter school students scoring in the range of New England states. Arizona charter schools serve a majority minority student population and spend only $8,041 per student- about a thousand less per student than Arizona districts and far less than the average spending in New England states.
Arizona’s AZ Merit exam demonstrated even larger gaps between charter and district scores than the NAEP, providing external validation for the NAEP scores.
Look at those guys! Their NAEP scores are going to collapse!
A little before the release of the 2015 NAEP, Mike Petrilli offered a friendly bet of a beer to me and Lisa Graham Keegan that Arizona’s NAEP scores would go down between 2013 and 2015 based on economic difficulties. We both instinctively thought they would go up, and they did. We are thirsty Mike! Taking a longer term view of the entire Great Recession period however proves more revealing.
Arizona scores have improved at six times the national rate on 4th grade math, 7 times the national rate on 8th grade math, five times the rate on 4th grade reading and 2.67 times the rate on 8th grade reading. How did a state that saw a decline in inflation adjusted spending per pupil drop from $9,438 in 2007 to $7,828 in 2014 (see JLBC doc link above) manage to outpace the nation in progress by such a wide margin? District interests here have a non-stop mantra about Arizona’s relevantly low ranking in per pupils funding but, er, why are we outpacing the nation by such a wide margin even as our funding declines?
Whoa- that’s unpossible!
Something is wrong here- but it is not Arizona’s positive score trends. What is wrong is some very common assumptions about K-12. I’ll get to that below.
The reality of Arizona K-12 improvement is of course complicated and defies any single explanation, with big changes going on at the same time. One factor that obviously contributed and that we can quantify charter schools. The next figure shows the NAEP gains by subject/grade for Arizona students for districts and charters (2015 scores minus 2007 scores).
Some may attempt to dismiss the difference between charters and districts as a product of differences in student populations. Only a random assignment study could definitively test this assumption, but a large amount of evidence suggests which way such a (sadly non-existent) study would fall. Arizona charter students rank well when compared to statewide averages when compared to a wide variety of subgroups (general ed, White, Hispanic, etc.) While differences in student populations could explain some of the differences between Arizona charters and Arizona districts, they can’t be put to similar use in explaining why Arizona students outscore similar students in New Hampshire. Arizona law also require random lottery admissions, serve a majority-minority student population and the improvement we see in the district scores does not exactly sit comfortably with a massive brain drain to charters story. If all of Arizona’s brightest students were fleeing to charters, it would put a substantial drag on district scores. Instead we see district scores improving.
Arizona has a higher percentage of students attending charters than any other states, but that still only falls in the teens– 13.9% in 2012-13. Even so these gains are large enough to make a noticeable difference the aggregates:
The reason I was willing to take Mike’s friendly bet on 2015 NAEP scores- I believe that by far the greatest opportunity to improve K-12 lies in making more efficient use of existing resources. In the opening pages of his 2004 book Hard America Soft America: Competition vs. Coddling and the Battle for the Nation’s Future the astute observer Michael Barone noted the following:
Public schools for example may be the most notable example of a predominantly Soft institution-which helps explain why American children are confined mostly to Soft America. But as we will see, our schools have not always been so Soft; they have contained corners of Hardness, and there are signs they are getting Harder now.
“Coddling” is not a term one would use to describe Arizona public education during the 2007-2015 period. Declining spending forced both district and charter leaders to seek efficiency. The state passed a law forbidding schools from making reduction in force decisions exclusively on length of service- this was very wise. Ineffective/expensive workers should be the first to go in a reduction in force- the alternative being to RIF a much larger number of young employees regardless of their effectiveness. Federal stimulus and a temporary sales tax increase delayed the need for these adjustments-but only temporarily. During this period Arizona lawmakers began grading schools A-F, and the combination of (mostly) recession related slow population growth and expanded competition halted what had been a non-stop process of student population growth for districts. Charters continue to gain market share against districts- and now both a more rigorous state test and NAEP show a substantial academic advantage for charter students.
None of this is easy for district leaders. It’s not exactly the cold howling wind of market competition, but it is a much higher level of competition and transparency than that to which the K-12 folks feel accustomed. Their world has become less stable and more competitive-Harder to use Barone’s phrase. To their credit, many district leaders have embraced the challenge.
It’s very difficult. It’s also very good for children.
So half way through reading Matthew Ridley’s new book The Evolution of Everything I come across a perfect distillation of the background meta-narrative of the JPGB in the last paragraph of Chapter 13:
The elite gets things wrong, says Douglas Carswell in The End of Politics and the Birth of iDemocracy, ‘because they endlessly seek to govern by design in a world that is best organized spontaneously from below.’ Public policy failures stem from planners excessive faith in deliberate design. ‘They constantly underrate the merits of spontaneous, organic arrangement, and fail to recognize that the best plan is often not to have one.’
Out here in AZ- a central planner’s K-12 nightmare of Wild West chaos-since the bust of 2007 we’ve had NAEP gains between 2.5 to 7 times greater than the national average. What passes for a consensus on Newtonian mechanics back east will struggle to explain this- something is wrong with the orbit of Mercury! Best perhaps to ignore outcomes entirely- as the New York Times did in an article this summer about AZ K-12:
Arizona in particular has been crippled by several years of targeted cuts at the state level and local voters’ repeated refusals to raise property taxes to offset these shortfalls.
Thank you for your touching concern NYT, but I’m feeling pretty handi-capable about right now. I know states with budget cuts and dysfunction in central command are not supposed to make nationally enviable academic progress during a very trying period. We alas didn’t get the memo out here in the patch so we went and did it anyway.
Posted by matthewladner 
Jay P. Greene's Blog 