Drowning in Dollars but Starving for Gains: DCPS Spending and Scores in Context

February 10, 2016

 

(Guest Post by Matthew Ladner)

In our next exciting episode of reviewing the DC education scene from the new study I wrote with Heritage we take a look at where the District remains despite a stunning level of investment and 25 years of improvement.

So let’s put DC into context in terms of revenue per pupil with data from the United States Census Bureau.

Ladner DC 1

DC charter schools seem to be bringing in somewhere in the neighborhood of half of that figure on a per pupil basis. Now let’s see what the Trial Urban District Assessment NAEP has to say about how DC kids compare. The chart below compares kids on NAEP 8th grade math scores for students whose parents graduated from high-school but did not attend college. The hope here is to rank districts by kids not born on third base.

Ladner DC 5

Note that if you do the same comparison by FRL status instead of parental education you still find DC ranked only ahead of Detroit and behind everyone else. The 17 point advantage for DC charter school kids in the above chart is considerable, but as the comparison makes painfully obvious, DC charters may be on their way, but they have not arrived. Still with less money and better scores the ROI is far, far higher than DCPS.

The heartbreaking part of the story however lies with the DCPS students. I’m not going to bother to look up the revenue per pupil statistics for Detroit but I am putting the over/under at half that of DC. Decades into DC reform efforts DCPS remains largely unchanged- far better at spending money than at teaching children, other than those who bought or worked their way into the high performing schools.

Next episode we’ll discuss what to do next. It shouldn’t involve continuing to bang our heads against the “better scores through improved management in an utterly broken system” wall. It also does not involve giving up.

 


Arizona Gains and the Orbit of Mercury-Wrecking Balls for Flawed Paradigms

December 21, 2015

(Guest Post by Matthew Ladner)

Recently I made an off-hand comment about Arizona NAEP gains being like the problem with the orbit of Mercury. I decided that it would profit from some further explanation. Newtonian mechanics seemed to have everything figured out, with that nagging problem of the orbit of Mercury doing something it shouldn’t. The “problem” with the orbit of Mercury of course wasn’t really a problem at all. It turned out to be a problem with our incomplete understanding of how the universe works- as illustrated in the above video.

Arizona 8m NAEP

So just how do Arizona NAEP scores resemble the orbit of Mercury? The 2015 NAEP shows that Arizona charter school students scoring in the range of New England states. Arizona charter schools serve a majority minority student population and spend only $8,041 per student- about a thousand less per student than Arizona districts and far less than the average spending in New England states.

AZ charter vs. district

Arizona’s AZ Merit exam demonstrated even larger gaps between charter and district scores than the NAEP, providing external validation for the NAEP scores.

Moreover Arizona charters, like districts, have been operating in a very tough environment. Back in my Goldwater Institute days, I would go to the Joint Legislative Budget Committee website as a rich source of for refuting complaints about public school spending. As it happened, the districts were getting more students, and the spending per pupil was climbing. Cry me a river. The Great Recession however clobbered the housing dependent Arizona economy and the same document now shows that the Great Recession shot Arizona school funding in the arm, punched the gunshot wound, and then threw us out through the front windshield.

Look at those guys! Their NAEP scores are going to collapse!

A little before the release of the 2015 NAEP, Mike Petrilli offered a friendly bet of a beer to me and Lisa Graham Keegan that Arizona’s NAEP scores would go down between 2013 and 2015 based on economic difficulties. We both instinctively thought they would go up, and they did. We are thirsty Mike! Taking a longer term view of the entire Great Recession period however proves more revealing.

AZ Mercury 1

Arizona scores have improved at six times the national rate on 4th grade math, 7 times the national rate on 8th grade math, five times the rate on 4th grade reading and 2.67 times the rate on 8th grade reading. How did a state that saw a decline in inflation adjusted spending per pupil drop from $9,438 in 2007 to $7,828 in 2014 (see JLBC doc link above) manage to outpace the nation in progress by such a wide margin? District interests here have a non-stop mantra about Arizona’s relevantly low ranking in per pupils funding but, er, why are we outpacing the nation by such a wide margin even as our funding declines?

Whoa- that’s unpossible!

Something is wrong here- but it is not Arizona’s positive score trends. What is wrong is some very common assumptions about K-12. I’ll get to that below.

The reality of Arizona K-12 improvement is of course complicated and defies any single explanation, with big changes going on at the same time. One factor that obviously contributed and that we can quantify charter schools. The next figure shows the NAEP gains by subject/grade for Arizona students for districts and charters (2015 scores minus 2007 scores).

Arizona Mercury 2

Some may attempt to dismiss the difference between charters and districts as a product of differences in student populations. Only a random assignment study could definitively test this assumption, but a large amount of evidence suggests which way such a (sadly non-existent) study would fall. Arizona charter students rank well when compared to statewide averages when compared to a wide variety of subgroups (general ed, White, Hispanic, etc.) While differences in student populations could explain some of the differences between Arizona charters and Arizona districts, they can’t be put to similar use in explaining why Arizona students outscore similar students in New Hampshire. Arizona law also require random lottery admissions, serve a majority-minority student population and the improvement we see in the district scores does not exactly sit comfortably with a massive brain drain to charters story. If all of Arizona’s brightest students were fleeing to charters, it would put a substantial drag on district scores. Instead we see district scores improving.

Arizona has a higher percentage of students attending charters than any other states, but that still only falls in the teens– 13.9% in 2012-13. Even so these gains are large enough to make a noticeable difference the aggregates:

Arizona Mercury 3

The differences in the above charts only display direct impact of charter school scores the statewide average. We have substantial reason to believe however that the growth of charter schools has indirectly raised Arizona scores as well through competition. In other words, charter schools almost certainly deserve some of the credit for the blue columns in the above chart rather than merely the difference between the red and the blue.

The reason I was willing to take Mike’s friendly bet on 2015 NAEP scores- I believe that by far the greatest opportunity to improve K-12 lies in making more efficient use of existing resources. In the opening pages of his 2004 book Hard America Soft America: Competition vs. Coddling and the Battle for the Nation’s Future the astute observer Michael Barone noted the following:

Public schools for example may be the most notable example of a predominantly Soft institution-which helps explain why American children are confined mostly to Soft America. But as we will see, our schools have not always been so Soft; they have contained corners of Hardness, and there are signs they are getting Harder now.

“Coddling” is not a term one would use to describe Arizona public education during the 2007-2015 period. Declining spending forced both district and charter leaders to seek efficiency. The state passed a law forbidding schools from making reduction in force decisions exclusively on length of service- this was very wise. Ineffective/expensive workers should be the first to go in a reduction in force- the alternative being to RIF a much larger number of young employees regardless of their effectiveness. Federal stimulus and a temporary sales tax increase delayed the need for these adjustments-but only temporarily. During this period Arizona lawmakers began grading schools A-F, and the combination of (mostly) recession related slow population growth and expanded competition halted what had been a non-stop process of student population growth for districts. Charters continue to gain market share against districts- and now both a more rigorous state test and NAEP show a substantial academic advantage for charter students.

None of this is easy for district leaders. It’s not exactly the cold howling wind of market competition, but it is a much higher level of competition and transparency than that to which the K-12 folks feel accustomed. Their world has become less stable and more competitive-Harder to use Barone’s phrase. To their credit, many district leaders have embraced the challenge.

It’s very difficult. It’s also very good for children. 

 


Best Organized Spontaneously from Below

December 15, 2015

(Guest Post by Matthew Ladner)

So half way through reading Matthew Ridley’s new book The Evolution of Everything I come across a perfect distillation of the background meta-narrative of the JPGB in the last paragraph of Chapter 13:

The elite gets things wrong, says Douglas Carswell in The End of Politics and the Birth of iDemocracy, ‘because they endlessly seek to govern by design in a world that is best organized spontaneously from below.’ Public policy failures stem from planners excessive faith in deliberate design. ‘They constantly underrate the merits of spontaneous, organic arrangement, and fail to recognize that the best plan is often not to have one.’

Out here in AZ- a central planner’s K-12 nightmare of Wild West chaos-since the bust of 2007 we’ve had NAEP gains between 2.5 to 7 times greater than the national average. What passes for a consensus on Newtonian mechanics back east will struggle to explain this- something is wrong with the orbit of Mercury! Best perhaps to ignore outcomes entirely- as the New York Times did in an article this summer about AZ K-12:

Arizona in particular has been crippled by several years of targeted cuts at the state level and local voters’ repeated refusals to raise property taxes to offset these shortfalls.

Thank you for your  touching concern NYT, but I’m feeling pretty handi-capable about right now. I know states with budget cuts and dysfunction in central command are not supposed to make nationally enviable academic progress during a very trying period. We alas didn’t get the memo out here in the patch so we went and did it anyway.

Oh and by the way Arizona style austerity of the 2007-2015 variety may make a nasty appearance in a state near you in the near future. Feel free to pop out and take some notes on how to thrive through it- and yes you can bring your golf clubs.

 

 

 

 

 


CeleNAEP for AZ Charters continues in AZ Merit Scores

December 4, 2015

(Guest Post by Matthew Ladner)

Laissez le bon temps rouler! The AZ Department of Education released results of the new AZ Merit test, and the Arizona Charter School Association released a breakdown of the scores showing that AZ charter school students scored 5%-12% higher scores depending upon the grade level/subject (see below).

Sadly the AZ Merit data comes in the form of an excel spreadsheet rather than a NAEP-like data slicer, so a control for differences in student demographics, program status, etc. cannot be readily performed. Some of the differences in scores between AZ charters and districts are likely to be explained by student differences. When you examine the NAEP, it is also the case that some of the differences between New England states and a state like Arizona will also be explained by student differences. No one I know however doubts that the average public school in MA outscores the average public school in AZ.

AZ Merit aligns much more closely to NAEP in terms of student results than the jumped the shark over a decade ago AIMS test. The differences observed in raw scores are generally larger than those existing between the top NAEP state (MA) and Arizona. The AZ Merit scores therefore reinforce the NAEP data’s finding that Arizona charter schools have something special cooking out in the cactus patch.

Korea- your scores are next!


Slice and Dice the Data but Arizona Charters Continue to CeleNAEP Good Times

November 16, 2015

(Guest Post by Matthew Ladner)

So I’ve still been digging into the AZ NAEP data. When something seems too good to be true, it is best to assume it isn’t true. I decided to investigate the possibility that something goofy was going on with the free and reduced lunch variable. Not all  Arizona charters choose not to participate in the program, and the eligibility criteria for the program has changed over time.

Parental education may be a good stand-in for what may be a suspect income variable. At the 8th grade level, the NAEP data slicer has an a variable for parental education. The below figure presents the 8th grade math scores for students with college graduate parents. In order to account for possible differences in special program participation, the figure is only for general education students with college graduate parents (the ranking results don’t change much if you look at all students). I will again stress that these comparisons do not substitute for a proper random assignment study-only that they tell us more than an examination of aggregate scores for all students.

NAEP AZ charter 8m parent educ

Watch out New England…Arizona charter schools are coming to get you!

For you incurable skeptics, the below figure presents the same comparison using 8th grade reading, and bear in mind that each NAEP test involves a different sample of students.

NAEP AZ charter 8r parent educ

We can also look at these numbers by race/ethnicity. NAEP provides subset numbers for Anglos and Hispanics attending charter schools in Arizona. Here is the NAEP 8th grade reading test for Hispanic students:

AZ Charter 8r Hispanic

And here it is for Anglo students:

AZ Charter 2015 NAEP 8r Anglo

Okay but what if those Arizona charter schools are chock full of Anglo kids whose parents graduated college? Now the National Alliance of Public Charter Schools reports that Arizona charter schools have a majority-minority student body, so this is not the case- but what if a large portion of the Anglo kids attending charter schools have parents with college degrees? Ok, well, let’s compare Anglo kids whose parents graduated from college:

AZ Charter 2015 NAEP 8r Anglo college

Did I mention the part where Arizona charter schools did this with $8,041 per kid in public funding? Better results at a lower cost is what America is going to need very soon- and well here it is. Massachusetts NAEP scores taste like chicken btw, only gamier, could use a little salt.

 


CeleNAEP Good Times-C’MON!

November 5, 2015

(Guest Post by Matthew Ladner)

My #Shadowfaction partner in crime Lisa Graham Keegan and I hit the pages of the Arizona Republic to celeNAEP good times in Arizona. Teaser trailer:

NAEP has given 40 exams on mathematics and reading to representative samples of Arizona students in various years since the early 1990s. As a low-income state with more than its share of student challenges (high poverty and non-native English speaking rates), Arizona has never met or exceeded the national average score – until now.

Arizona’s eighth-graders edged out the national average in mathematics and only narrowly missed doing the same in reading. Therefore, Arizona’s Class of 2019 carries a special distinction in state history – one that future classes can both match and exceed. Achievement has improved substantially since the last pre-recession measures in 2007.

Of notable attention are Arizona’s charter-school students, who matched the scores for the highest-scoring states on the 2015 NAEP. On eighth grade mathematics, for instance, Arizona charter students scored in a statistical dead heat with Massachusetts, the highest scoring of the 50 states.

 


Arizona charter schools rocked the 2015 NAEP so hard that…

November 4, 2015

(Guest Post by Matthew Ladner)

,,,they scored the same as the nationwide number for private school students on the NAEP 8th grade math test in 2013 and 2011 (no new private school number in 2015). I’m not sure what to say about this other than:


Follow

Get every new post delivered to your Inbox.

Join 3,046 other followers