The Next Accountability

July 6, 2016


(Guest post by Greg Forster)

Today the Friedman Foundation carries the introduction to my forthcoming series, The Next Accountability. In this series I will make the case that the education reform movement is being pulled apart by differing visions of “accountability”:

Our forefathers built the education reform movement on a foundation that all reformers shared: We need to hold schools accountable, so they’ll give kids the education we want them to get. Now we’re discovering cracks in the foundation. It turns out we don’t agree on what we want, or on how we get schools to deliver it.

These differing visions of accountablity are really differing visions of what education is for – which are in turn differing visions of what constitutes a good human life:

A few of us, however, think that all this technocracy is precisely what we have been fighting against all along. It is essentially an extension of the old regime’s philosophy: We’re the education experts, and we know best! It’s just as impersonal and unresponsive to the real needs of real people as the blob. It’s as if we defeated the Soviet Union, and then celebrated our victory by imposing communism on Western Europe and North America.

For this reason, supporters of choice are going to have to get beyond talking points and canned rhetoric about “markets” and “competition”:

America needs to rethink what we really want from schools. Whether of the old or new variety, technocratic systems fail not only because they can be manipulated by greedy and incompetent people or because they lack sufficient information about client preferences (although these things are also worth remembering)—technocracy fails more importantly because it is based on a wrong understanding of what education is for.

Knowing what we want requires us to reawaken to who we are. All the great thinkers who have cast big visions for education, from Plato and Aristotle to Aquinas and Locke to Rousseau and Dewey, agreed that knowing something about what it means to educate children ultimately requires us to know something about what it means to be human. If opponents of technocracy can’t say something about that, all our rhetoric about markets and competition is chaff in the wind.

In the forthcoming articles in this series, I will lay out a vision of what we want from schools, and how we can create new approaches to accountability that will help us get what we want, without falling into the technocratic trap.

Follow the link and read the introduction to get a sneak preview of what that will entail.

Championing a different vision of accountability may increase the current conflicts in the education reform movement rather than decrease them. Technocracy is rooted in a vision of what education is for – of what is good for human beings – that many people in the education reform movement really believe in. We shouldn’t expect them to give it up without a fight.

But, as I will argue, a fresh vision can attract new allies. In many cases they will be more powerful allies than our current technocratic coalition partners, who are increasingly dismissive of us and our concerns anyway. And if we’re not willing to fight technocracy, even when it comes from our friends (in many cases former friends, who turned on us as soon as they thought they could) what was the fight against the blob all about in the first place?

It is only through that kind of conflict that really historic progress gets made. Dr. Martin Luther King Jr. was not interested in renegotiating the terms of segregation. He came not to reform it but to exterminate it. We too must fight not for a renegotiation of terms with the technocratic beast, but for its end.

The remainder of this series will launch shortly after Friedman Legacy Day on July 29, when the Friedman Foundation will be making big announcements about its future. One thing I can promise you: we will never leave behind the mission to fight for justice and freedom that has always animated the foundation, which is Milton’s greatest legacy to us.

In the meantime, I welcome your feedback and look forward to the next stage of the fight!

Liberty and Justice for All

June 1, 2016

(Guest Post by Matthew Ladner)

Kingsland writes a hilarious post on five theories of change in K-12 philanthropy.  Over on the Ed Next podcast, Paul Peterson explains why he believes top-down command and control improvement strategies have jumped the shark and calls for a renewed focus on choice as an improvement strategy.


Overregulation Is All You Need: A Response to Paul Bruno

February 29, 2016


(Guest Post by Jason Bedrick)

Over at the Brookings Institution’s education blog, Paul Bruno offers a thoughtful critique of  Overregulation Theory (OT), the idea that government regulations on school choice programs can undermine their positive effects. Bruno argues that although OT is “one of the most plausible explanations” of the negative results that two studies of Louisiana’s voucher program recently found, it is not “entirely consistent with the available evidence” and “does not by itself explain substantial negative effects from vouchers.”

I agree with Bruno–and have stated repeatedly–that the studies’ findings do not conclusively prove OT. That said, I believe both that OT is consistent with the available evidence and that it could explain the substantial negative effects (though I think it’s likely there are other factors at play as well). I’ll explain why below, but first, a shameless plug:

On Friday, March 4th at noon, the Cato Institute will be hosting a debate over the impact of regulations on school choice programs featuring Patrick Wolf, Douglas Harris, Michael Petrilli, and yours truly, moderated by Neal McCluskey. If you’re in the D.C. area, please RSVP at this link and join us! Come for the policy discussion, stay for the sponsored lunch!

Is the evidence consistent with Overregulation Theory?

Bruno notes that the differences in enrollment trends between participating and non-participating private schools is consistent with OT. Participating schools had been experiencing declining enrollment in the decade before the voucher program was enacted whereas non-participating schools had slightly increasing enrollment on average. This is consistent with the OT’s prediction that better schools (which were able to maintain their enrollment or grow) would be more likely eschew the vouchers due to the significant regulatory burden, while the lower-performing schools (which were losing students) were more desperate for students and funding, and were therefore more willing to jump through the voucher program’s regulatory hoops. However, Bruno calls this evidence into question:

For one thing, the authors of the Louisiana study specifically check to see if learning outcomes vary significantly between schools experiencing greater or lesser prior enrollment declines, and find that they do not. (Bedrick acknowledges this, but doubts there was enough variation in the enrollment trends of participating schools to identify differences.)

We should be skeptical of the explanatory value of the study’s enrollment check. There is no good reason to assume that the correlation between enrollment growth or decline among the small sample of participating schools (which had significantly negative growth, on average) is the same as among all private schools in the state. Making such an assumption is like a blind man holding onto the truck of an elephant and assuming that he’s holding a snake.

The study does not show the variation in enrollment trends among the participating and non-participating schools, but we could imagine a scenario where the enrollment trend among participating schools ranged, say, from -25% to +5% while the range at non-participating schools was -5% to +25%. As shown in the following charts (which use hypothetical data), there may be a strong correlation between enrollment trends and outcomes among the entire population, while there is little correlation in the subset of participating schools.

Enrollment Growth and Performance, Participating Private Schools (Hypothetical)

Screen Shot 2016-02-29 at 4.57.05 PM

Enrollment Growth and Performance, All Private Schools (Hypothetical)

Screen Shot 2016-02-29 at 10.05.29 PM.png

In short, looking at the relationship between enrollment growth and performance in the narrow subset of participating schools doesn’t necessarily tell us anything about the relationship between enrollment growth and performance generally. Hence the study’s “check” that Bruno cites does not provide evidence against OT.

Is there evidence that regulations improve performance?

Bruno also cites evidence that regulations can have a positive impact on student outcomes:

Joshua Cowen of Michigan State University also points out that there is previous evidence of positive effects from accountability rules on voucher program outcomes in other states (though regulations may differ in Louisiana).

The Cowen article considers the impact of high-stakes testing imposed on the Milwaukee voucher program during a multi-year study of that program. The “results indicate substantial growth for voucher students in the first high-stakes testing year, particularly in mathematics, and for students with higher levels of earlier academic achievement.” But is this strong evidence that regulations improve performance? One of the authors of both the original Milwaukee study and the cited article–JPG all-star, Patrick Wolf–cautions against over-interpreting these results:

Ours is one study of what happened in one year for one school choice program that switched from low-stakes testing to high-stakes testing.  As we point out in the report, it is entirely possible that the surge in the test scores of the voucher students was a “one-off” due to a greater focus of the voucher schools on test preparation and test-taking strategies that year.  In other words, by taking the standardized testing seriously in that final year, the schools simply may have produced a truer measure of student’s actual (better) performance all along, not necessarily a signal that they actually learned a lot more in the one year under the new accountability regime.

If we had had another year to examine the trend in scores in our study we might have been able to tease out a possible test-prep bump from an effect of actually higher rates of learning due to accountability.  Our research mandate ended in 2010-11, sadly, and we had to leave it there – a finding that is enticing and suggestive but hardly conclusive.

It’s certainly possible that the high-stakes test improved actual learning. But it’s also possible–and, I would argue, more probable–that changing the stakes just meant that the schools responded to the new incentive by focusing more on test-taking strategies to boost their scores.

For that matter, even if it were true that the regulations actually improved student learning, that does not contradict Overregulation Theory. Both advocates and skeptics of the regulations believe that schools respond to incentives. Those of us who are concerned about the impact of the regulations don’t believe that they can’t improve performance. Rather, our concern is that regulations imposed from above are less effective at improving performance than the incentives created by direct accountability to parents in a robust market in education, and may have adverse unintended consequences.

To explain: We’re concerned that regulations forbidding the use of a school’s preferred admissions standards or requiring the state test (which is aligned to the state curriculum) might drive away better-performing schools, leaving parents to choose only among the lower-performing schools. We’re concerned that price controls will inhibit growth, providing schools with an incentive only to fill empty seats rather than to scale up. We’re concerned that mandatory state tests will inhibit innovation and induce conformity. None of these concerns rule out the possibility (or, indeed, the likelihood) that over time, requiring private schools to administer the state test and report the results and/or face sanctions based on test performance will improve the participating schools’ performance on that test.

Again: we agree that schools respond to incentives. We just think the results of top-down incentives are likely to be inferior to the results of bottom-up choice and competition, which have proved to be powerful tools in so many other fields for spurring innovation and improving quality.

Can Overregulation Theory alone explain the negative results in Louisiana?

Finally, Bruno questions whether OT alone explains the Louisiana results:

[E]ven if regulation prevented all but the worst private schools from participating, this would explain why students did not benefit from transferring into them, but not why students would transfer into them in the first place.

So Overregulation Theory might be part of the story in explaining negative voucher effects in Louisiana, but it is not by itself sufficient. To explain the results we see in the study, it is necessary to tell an additional story about why families would sort into these apparently inferior schools.

Bruno offers a few possible stories–that parents select schools “that provide unobserved benefits,” that the voucher program “induced families to select inferior schools,” or that parents merely “assume any private school must be superior to their available public schools”–but any of these can be consistent with OT.  Indeed, the second story Bruno offers is practically an extension of OT: if the voucher regulations truncate supply so that it is dominated by low-quality schools, and the government gives false assurances that they have vetted those schools, then it is likely that we will see parents lured into choosing inferior schools.

That’s not to say that there are no other factors causing the negative results. It’s likely that there are. (I find Douglas Harris’s argument that the private schools’ curricula did not align with the state test in the first year particularly compelling, though I don’t think it entirely explains the magnitude of the negative results.) We just don’t have any compelling evidence that OT is wrong, and OT can suffice to explain the negative results.

I will conclude as I began: expressing agreement. I concur with Bruno’s assessment that “it is likely that the existing evidence will not allow us to fully adjudicate between competing hypotheses.” Indeed, it’s likely that future evidence won’t be conclusive either (it rarely is), but I hope that further research will shed more light on this important question. Bruno concludes by calling for greater efforts to “understand how families determine where their children will be educated,” noting that by understanding how and why parents might make “sub-optimal — or even harmful” decisions will help “maximize the benefits of school choice while mitigating its risks.” These are noble goals and I share Bruno’s desire to pursue them. I just hope that policymakers will approach what we learn with a spirit of humility about what they can accomplish.

Cool Honesty Gap Graphics on Truth in Advertising in State Testing

January 28, 2016

(Guest Post by Matthew Ladner)

So yesterday the Godfather of the School Choice movement presented evidence that nuked an oft-repeated claim regarding the great dummy down of American testing. Today the folks at Honesty Gap mop up the poor irradiated wretches to put them out of their misery with some cool graphics showing the increased alignment between state tests and NAEP like:



Now for people like me who support Arizona developing a set of standards unique to Arizona and replacing the current CC standards and possibly test so long as they are as good or better, the presence of Oklahoma on that last chart is a problem. For anti-CC crusaders, it’s an even a larger problem, unless of course you are just shameless is your support for tests that a chimp could sometimes pass blindfolded. Yes Tex I am looking at you. A constructive vote of no-confidence remains a respectable path to leaving CC in the rear view mirror, so I hope Oklahoma will pull it off in plenty of time for the 2017 NAEP.

So about those “Great Dummy Down” Claims…

January 27, 2016

(Guest Post by Matthew Ladner)

Let me take a moment to reiterate that in my home state I’m very comfortable with Governor Ducey’s goal to create a set of high academic standards unique to Arizona. I see little value in “common” standards (NAEP scratches my cross-state comparison itch) but I hate state tests that the Wall Street stock picking chicken could pass on his way to receiving a false state endorsement of “proficiency” with the burning hatred of a thousand suns. I have no idea where this will ultimately wind up and I can easily imagine better strategies than those adopted, etc.

Having said that, let’s just say that the “great dummy down” claim just got put on the shelf next to bus-based retinal scan stories and United Nations conspiracy theories:

A Harsh Apology to the Arizona Class of 2006 and the True Meaning of Accountability

September 2, 2015

(Guest Post by Matthew Ladner)

I had the chance to make a presentation on testing and private choice programs recently, and received a request to share a few slides from that presentation here on JPGB. So the first is from Arizona, circa 2006. I chose 2006 because we have a study that follows the entire public school Class of 2006 through the higher education system.


So for those of you squinting at your Ipad- the columns read: Kids attending AZ public schools taking state math and reading tests (100%!), AZ Class of 2006 who read proficiently as 8th graders on the 2002 NAEP 8th grade reading exam (errr, 23%), Percent of Class of 2006 graduating class who went on to earn a Bachelor degree by the end of 2012 (errr 18.6%) and finally the percent of Arizona public schools who earned an “Underperforming” or “Failing” label in 2006 (*cough* 6.5%).

So who was held “accountable” in this slide. Not the Governor she was reelected by a wide margin in 2006. Oh what about the Superintendent of Public Instruction? Nope- he was reelected as well. Did Arizona have a mass culling of ineffective school superintendents in 2006? What about teachers? Nope and nope- it was business as usual.

Let’s compare the accountability for the staff at the 6.5% of schools who received a nice-so-nice label compared to that of the students. Now that it is 2015, what is the chance that any of the adults in those 6.5% of schools carry around a nine-year old label around with them as a burden, even if they remain in education and are still remain employed at the same school? Right- now what about the 81.4 percent of the Class of 2006 who either never attended college or who were among the waves of people who dropped out of college in debt with little to show for it?

The latter scenario constitutes a much harsher form of accountability than Arizona’s former “whip truly terrible schools with a wet noodle accountability.”  Sorry Class of 2006- I know that the state of Arizona gave you the equivalent of a Good Housekeeping Seal of Approval based on your results on a laughably simple AIMS test, then they kicked the can down the road on an high-school exit exam, then they labelled your school “performing” when it was actually anything but academically performing. That “performing” actually meant that it was performing its duty to employ adults and might occasionally facilitate learning if a student had a deep motivation to do it on their own. You were very foolish not to clue into this and now you will have to pay for it.

We the taxpayers and adult policymakers of Arizona feel some regret about all of this Class of 2006 for all these things, but ultimately you should have ignored the fact that the adults in your life were telling you everything was fine, and you should have studied harder especially when you were learning to read in K-3, since 77 percent of you failed to reach proficiency as 8th graders. It was really absurd so many of you thought you could do college level work, but the universities disabused you of that notion quickly didn’t they? Don’t worry the Class of 2007 temporarily filled your spots after you dropped out.

In the next life you should not be so trusting of adults and study harder. Perhaps you will take this lesson to heart as parents.  Go suffer the consequences of your actions and just think of how much worse this might have turned out if Arizona did not hold school systems accountable. The greatest trick the public school lobby ever pulled was convincing the world that the publication of scores on minimal skills math and reading tests constituted “accountability.” And like that **poof**

…meaningful accountability was gone, unless by “accountability” we mean watching helplessly as students suffer the long-term consequences for failing to acquire competitive skills.

Now as a post-script, things have improved somewhat since 2006 in Arizona. Instead of handing out “performing” labels, the state uses letter grades. Grades of C or D are closer to truth in advertising than “Performing.” The wretched AIMS test has finally received the mercy killing it so richly deserved.  Sorry I-hate-CC-with-a-purple-passion tribe, the new test aligns much closer to Arizona’s performance on NAEP so it represents an undeniable upgrade over AIMS, at least so far. Yes, it could have been accomplished by other means etc. etc. but the sad reality is that we sat around indifferently for years as the fraud described above played out.  My humble suggestion at this point would be to offer constructive and rigorous counter proposals to AZ Merit because I hope that if you’ve reached this part of the post you’ll at least acknowledge the true horror of the AIMS regime. I mean it was cooked up by a group of Arizona teachers in 1994, which makes it near sacred and all, but that can’t make up for the system being horribly mismanaged by the AZ Department of Ed and State Board of Education after that. It devolved into a cruel joke on children.

Yes Jay I get it they probably will do the same with the new test sooner or later.  How long do you expect this peace to last?

…as long as it can.

In the end, this too shall pass, so the most enduring accountability going on in Arizona today involves parental choice. Parental choice in fact represents the ultimate form of accountability that no system of aggregate test scores and school labels can ever replace.  Even at its best such accountability is an aggregate phenomenon, whereas parental choice allows parents to hold schools responsible at the individual level by voting with their feet.

Since 2006, AZ charter schools have reached a more meaningful scale. Arizona now has the highest percentage of students attending charter schools (almost 18%) of any state. Parents have used their contacts and Greatschools to figure out that even their allegedly swell schools leave much to be desired and have commenced to pounding on the doors of high quality charter operators, developing huge waiting lists.

The highest rated general enrollment school in the Arizona Board of Regents analysis of higher education outcomes Tempe Prep- was the forerunner of the Great Hearts system of schools that now has 22 campuses and mile long waiting lists. These schools did not appear in the 2006 analysis because they either had not opened yet or did not have a senior class by 2006. Stay tuned-the Board of Regents will soon have an updated analysis. Our private choice programs in the aggregate are mostly helping private schools to remain viable against the rise of charters. We need to do more on that front, and we need to help high quality charters replicate.

Meanwhile, for the first time ever, Arizona districts find their enrollment in decline in absolute terms. Before the great recession charters and choice were simply taking the edge off of district enrollment growth. In the last couple of years there has been district enrollment declines. Enrollment growth will eventually reverse this, but for now the charters are  basically absorbing all of it and more. Oh by the way, while Arizona’s NAEP scores are not high, they were higher than they have ever been in 2013. Sweet are the uses of adversity…

Arizona’s growing choice sector has created a constituency and will not be dispatched as easily as the well-meaning but ultimately failed efforts of the AIMS regime. Keep hope alive!


Senators to Emperor Weingarten: What is thy bidding my master?

July 14, 2015

“If we confuse the Senate with populist rhetoric, they could become a powerful ally against state reform efforts…”

(Guest Post by Matthew Ladner)

Not content to let the House have all the fun in gutting state testing data and creating perverse incentives for schools to sweep their low-performing students under the rug, the Senate is getting ready to join in as well. The civil rights community is up in arms-they should be, and we should all be with them.

If you are nursing the hope that school officials would not do such a thing, let me direct you to the following table on the % of schools that actually wound up being including special education students as a part of their accountability subset in 2009-10:

So if you live in either Connecticut, Maine or Utah, then you can hang on to the hope that every child will be counted. If you live in a state education the other 98% of American students, it is time to wake up to the fact that school officials have a long and predictable history of following the path where perverse incentives lead them, and don’t tend to let little things like the interests of children bother them overly much. Given the opportunity to make use of a “parental” opt-out, it is blindingly obvious that school officials will take full advantage of the provision to make themselves look good, just as they have used every available loophole to bury special education scores.

Some in the beltway likes to think that Congress is some sort of gathering of Olympians best positioned to guide the nation towards technocratic K-12 improvement. The House has already provided (additional) recent evidence to demonstrate this to be incredibly misguided, and the Senate seems poised to follow suit.  Anti-common core hysteria “The Devil Made Me Do It!” will not do for an excuse when one is contemplating wrecking state testing systems and creating a Freddie/Fannie level perverse incentive all in one fell swoop.

By the next time this law comes up for reauth some 13+ or so years hence, CC will likely be a distant memory. If you’ve been paying attention, states have been adopting their own tests left and right and they have control over their own cut scores. Oklahoma withdrew from the standards completely, and approximately nothing happened to them. Many states have begun a process to review and revise standards. The best case scenario is that states will choose to use something better than their old My Little Pony Book of Connect the Dots for their new tests, but it will be up to them in any case.

We would likely find a federal opt out of all criterion based tests not so easily dispatched. It would prove far more consequential if Hanushek and Loveless are to be believed. Once put into law, the unions will fiercely defend it, given that it completely thwarts the ability to consider test scores in tenure and retention decisions on the basis of criterion based tests. I would expect it to stay in place until the next reauth. States desiring to have campus level comparable data would have to create new systems to carry it out with non-criterion tests in an era of testing fatigue. Thanks DC!

None of this is likely to happen of course, given the high probability of a presidential veto. It grows ever more obvious however that the unions have outmanoeuvred reformers.

UPDATE: The Senate voted down the opt-out amendment most similar to the House Amendment 32-64. Faith in humanity (temporarily?) restored.