Overregulation Is All You Need: A Response to Paul Bruno

February 29, 2016

6-blind-men-hans

(Guest Post by Jason Bedrick)

Over at the Brookings Institution’s education blog, Paul Bruno offers a thoughtful critique of  Overregulation Theory (OT), the idea that government regulations on school choice programs can undermine their positive effects. Bruno argues that although OT is “one of the most plausible explanations” of the negative results that two studies of Louisiana’s voucher program recently found, it is not “entirely consistent with the available evidence” and “does not by itself explain substantial negative effects from vouchers.”

I agree with Bruno–and have stated repeatedly–that the studies’ findings do not conclusively prove OT. That said, I believe both that OT is consistent with the available evidence and that it could explain the substantial negative effects (though I think it’s likely there are other factors at play as well). I’ll explain why below, but first, a shameless plug:

On Friday, March 4th at noon, the Cato Institute will be hosting a debate over the impact of regulations on school choice programs featuring Patrick Wolf, Douglas Harris, Michael Petrilli, and yours truly, moderated by Neal McCluskey. If you’re in the D.C. area, please RSVP at this link and join us! Come for the policy discussion, stay for the sponsored lunch!

Is the evidence consistent with Overregulation Theory?

Bruno notes that the differences in enrollment trends between participating and non-participating private schools is consistent with OT. Participating schools had been experiencing declining enrollment in the decade before the voucher program was enacted whereas non-participating schools had slightly increasing enrollment on average. This is consistent with the OT’s prediction that better schools (which were able to maintain their enrollment or grow) would be more likely eschew the vouchers due to the significant regulatory burden, while the lower-performing schools (which were losing students) were more desperate for students and funding, and were therefore more willing to jump through the voucher program’s regulatory hoops. However, Bruno calls this evidence into question:

For one thing, the authors of the Louisiana study specifically check to see if learning outcomes vary significantly between schools experiencing greater or lesser prior enrollment declines, and find that they do not. (Bedrick acknowledges this, but doubts there was enough variation in the enrollment trends of participating schools to identify differences.)

We should be skeptical of the explanatory value of the study’s enrollment check. There is no good reason to assume that the correlation between enrollment growth or decline among the small sample of participating schools (which had significantly negative growth, on average) is the same as among all private schools in the state. Making such an assumption is like a blind man holding onto the truck of an elephant and assuming that he’s holding a snake.

The study does not show the variation in enrollment trends among the participating and non-participating schools, but we could imagine a scenario where the enrollment trend among participating schools ranged, say, from -25% to +5% while the range at non-participating schools was -5% to +25%. As shown in the following charts (which use hypothetical data), there may be a strong correlation between enrollment trends and outcomes among the entire population, while there is little correlation in the subset of participating schools.

Enrollment Growth and Performance, Participating Private Schools (Hypothetical)

Screen Shot 2016-02-29 at 4.57.05 PM

Enrollment Growth and Performance, All Private Schools (Hypothetical)

Screen Shot 2016-02-29 at 10.05.29 PM.png

In short, looking at the relationship between enrollment growth and performance in the narrow subset of participating schools doesn’t necessarily tell us anything about the relationship between enrollment growth and performance generally. Hence the study’s “check” that Bruno cites does not provide evidence against OT.

Is there evidence that regulations improve performance?

Bruno also cites evidence that regulations can have a positive impact on student outcomes:

Joshua Cowen of Michigan State University also points out that there is previous evidence of positive effects from accountability rules on voucher program outcomes in other states (though regulations may differ in Louisiana).

The Cowen article considers the impact of high-stakes testing imposed on the Milwaukee voucher program during a multi-year study of that program. The “results indicate substantial growth for voucher students in the first high-stakes testing year, particularly in mathematics, and for students with higher levels of earlier academic achievement.” But is this strong evidence that regulations improve performance? One of the authors of both the original Milwaukee study and the cited article–JPG all-star, Patrick Wolf–cautions against over-interpreting these results:

Ours is one study of what happened in one year for one school choice program that switched from low-stakes testing to high-stakes testing.  As we point out in the report, it is entirely possible that the surge in the test scores of the voucher students was a “one-off” due to a greater focus of the voucher schools on test preparation and test-taking strategies that year.  In other words, by taking the standardized testing seriously in that final year, the schools simply may have produced a truer measure of student’s actual (better) performance all along, not necessarily a signal that they actually learned a lot more in the one year under the new accountability regime.

If we had had another year to examine the trend in scores in our study we might have been able to tease out a possible test-prep bump from an effect of actually higher rates of learning due to accountability.  Our research mandate ended in 2010-11, sadly, and we had to leave it there – a finding that is enticing and suggestive but hardly conclusive.

It’s certainly possible that the high-stakes test improved actual learning. But it’s also possible–and, I would argue, more probable–that changing the stakes just meant that the schools responded to the new incentive by focusing more on test-taking strategies to boost their scores.

For that matter, even if it were true that the regulations actually improved student learning, that does not contradict Overregulation Theory. Both advocates and skeptics of the regulations believe that schools respond to incentives. Those of us who are concerned about the impact of the regulations don’t believe that they can’t improve performance. Rather, our concern is that regulations imposed from above are less effective at improving performance than the incentives created by direct accountability to parents in a robust market in education, and may have adverse unintended consequences.

To explain: We’re concerned that regulations forbidding the use of a school’s preferred admissions standards or requiring the state test (which is aligned to the state curriculum) might drive away better-performing schools, leaving parents to choose only among the lower-performing schools. We’re concerned that price controls will inhibit growth, providing schools with an incentive only to fill empty seats rather than to scale up. We’re concerned that mandatory state tests will inhibit innovation and induce conformity. None of these concerns rule out the possibility (or, indeed, the likelihood) that over time, requiring private schools to administer the state test and report the results and/or face sanctions based on test performance will improve the participating schools’ performance on that test.

Again: we agree that schools respond to incentives. We just think the results of top-down incentives are likely to be inferior to the results of bottom-up choice and competition, which have proved to be powerful tools in so many other fields for spurring innovation and improving quality.

Can Overregulation Theory alone explain the negative results in Louisiana?

Finally, Bruno questions whether OT alone explains the Louisiana results:

[E]ven if regulation prevented all but the worst private schools from participating, this would explain why students did not benefit from transferring into them, but not why students would transfer into them in the first place.

So Overregulation Theory might be part of the story in explaining negative voucher effects in Louisiana, but it is not by itself sufficient. To explain the results we see in the study, it is necessary to tell an additional story about why families would sort into these apparently inferior schools.

Bruno offers a few possible stories–that parents select schools “that provide unobserved benefits,” that the voucher program “induced families to select inferior schools,” or that parents merely “assume any private school must be superior to their available public schools”–but any of these can be consistent with OT.  Indeed, the second story Bruno offers is practically an extension of OT: if the voucher regulations truncate supply so that it is dominated by low-quality schools, and the government gives false assurances that they have vetted those schools, then it is likely that we will see parents lured into choosing inferior schools.

That’s not to say that there are no other factors causing the negative results. It’s likely that there are. (I find Douglas Harris’s argument that the private schools’ curricula did not align with the state test in the first year particularly compelling, though I don’t think it entirely explains the magnitude of the negative results.) We just don’t have any compelling evidence that OT is wrong, and OT can suffice to explain the negative results.

I will conclude as I began: expressing agreement. I concur with Bruno’s assessment that “it is likely that the existing evidence will not allow us to fully adjudicate between competing hypotheses.” Indeed, it’s likely that future evidence won’t be conclusive either (it rarely is), but I hope that further research will shed more light on this important question. Bruno concludes by calling for greater efforts to “understand how families determine where their children will be educated,” noting that by understanding how and why parents might make “sub-optimal — or even harmful” decisions will help “maximize the benefits of school choice while mitigating its risks.” These are noble goals and I share Bruno’s desire to pursue them. I just hope that policymakers will approach what we learn with a spirit of humility about what they can accomplish.


The Wisdom of the Market

July 17, 2015

Design vs Experience

(Guest Post by Jason Bedrick)

I’m humbled that Andy Smarick thinks I “offered the most compelling philosophical explanation for a system of choice” among the recent Fordham Institute Wonkathon participants. However, he misreads me when he states that my “professed ‘humility’ (we don’t know everything) came across as agnosticism (we can’t know anything) given that we’ve learned gobs about choice over twenty-five years.”

Nowhere do I claim “we can’t know anything.” Of course we do, and of course we can learn more. But questions remain: who are “we,” what do “we” know, and is that knowledge sufficient to achieve “our” ends?

By “we” Smarick seems to have in mind “policymakers and wonks” and the “what” that “we” supposedly know is that markets alone just don’t cut it so some very, very smart people must bend schools to their will impose government regulations to ensure accountability. To bolster his case, Smarick cites a recent article in National Affairs by Chester Finn and Bruno Manno on the lessons they’ve learned from their decades of experience studying charter schools:

Both strongly support school choice, but they concede the “vexing reality” that “market forces alone can’t reliably generate academic effectiveness.” Overconfidence led to accountability getting short shrift early. “Those present at the creation of the charter bargain (ourselves included) paid too little attention to how authorizing would work.”

Throughout the article, the [Finn and Manno] explain how events played out differently than expected. Because they assumed “a barely regulated marketplace would provide more quality control than it has…we focused on quantity rather than quality.” They were excited by policies that would spur “infusions of capital and entrepreneurialism,” but “we didn’t take seriously enough the risk of profiteering.”

Smarick claims that my Wonkathon entry’s “sanguine title, ‘Let the market work,’ runs headlong into Finn and Manno’s reflections.” As Justice Scalia might say: pure applesauce.

It may or may not be true that “market forces alone can’t reliably generate academic effectiveness” but Finn and Manno cannot draw that conclusion from experience in the charter sector because charters are not operating in a free market, never mind a “barely regulated” one. Charters are secular public schools that can’t charge families tuition, can’t devise their own criteria for admission, they have to meet certain state standards, and they can be shut down by their “authorizers” even if a sufficient number of students and parents prefer the charter to make it financially viable.

In other words, charters provide more choice and competition than the status quo, but charters are not operating in a market. The lack of a price mechanism alone should make that apparent. Drawing any conclusions about what an actual market in education would or would not produce based on the charter experience is ludicrous.

There’s some truth to Smarick’s contention that “theory without experience is [mere] intellectual play,” but he’s drawing the wrong lessons from Finn and Manno’s experience. Although it’s impossible to draw solid conclusions about a market from a non-market, the charter sector has much to teach policymakers about the chasm between policy intentions and policy results.

For example, Finn and Manno lament that “charters in many places are hobbled by many operational constraints, too little money, and, often, insufficient attention both to the delicate balance between quantity and quality.” These constraints often stem from the very regulatory framework that was intended to ensure quality. A 2010 Fordham study found that “state laws were the primary sources of constraint on charter school autonomy, accounting for three-quarters of the infringement that these schools experience.” This year, an American Enterprise Institute study found that “one-fourth of average charter application contains inappropriate and onerous requirements,” and that authorizes “sometimes mistake length for rigor” and “often prize innovation less than they say they do.”

The fatal conceit of the charter school agenda was that granting schools a bit more autonomy and granting parents a bit more choice in a controlled environment would create a true “market” in education. But a market requires a price mechanism, a means of channeling dispersed knowledge. Smarick accuses me of believing that “we can’t know anything” but that’s not so. Policymakers can’t possibly ever know enough to design the education system from the top down but the market can channel dispersed knowledge to produce higher quality through experimentation, evaluation, and evolution.

Smarick confuses the technical knowledge of experts for the dispersed knowledge of the entire system. Sure, technocrats have learned “gobs about choice” in a quarter-century, but they can’t possibly know enough to design the most effective possible education system. Likewise, a panel of a dozen Nobel-prize winning economists certainly knows “gobs” about how markets function, but they cannot possibly know enough to effectively set the price of tin on any given day.

The technocrats’ approach is attempt to define quality, measure it, and shut down any school that doesn’t live up to it — even against the will of parents. As Finn and Manno wrote:

Charter doctrine is clear: Bad schools should be closed (or “non-renewed”) by their authorizer. Yet it turns out to be as hard to close bad charters as traditional district schools. Hundreds of kids are affected, and the alternatives for them are often no better than the troubled charter. Furthermore, parents are almost universally hostile to the demise of their children’s current schools.

First, why are those parents “almost universally hostile” to closing down their chosen school? Could it be because “the alternatives for them are often no better” and probably worse? Could it be that the schools are effectively providing some things–safety, discipline, good values, a love of learning–that the parents legitimately prioritize over test scores in a few subjects?

Attempts to define and measure quality too often come at the cost of innovation. At present, states’ standardized testing regimes assume that all students should progress at the same pace across all subjects — a system that is anathema to reforms like competency-based learning which dispense with Carnegie units. Moreover, the focus on a few subjects both creates a perverse incentive for schools to focus on those subjects to the exclusion of others and overlooks the other, often more important areas where a school may be performing quite well. As AEI’s Michael McShane wrote:

Recently, I have been influenced by the work of Northwestern University economist Kirabo Jackson, whose fascinating NBER working paper calls into serious question policy’s recent overreliance on math and reading scores as the primary measure of the “goodness” of schools and teachers. As it turns out, teachers have important and measurable impacts on both the cognitive and non-cognitive development of students. While it’s certainly true that test scores can tell us something important about a teacher, what is troubling for the test-score types is that it looks like (1) non-cognitive scores are better predictors of later life success (completing high school, taking the SAT, and going to college) and (2) that it is not the same set of teachers that is good at raising both cognitive and non-cognitive measures.

Such has to be the same for schools, right? If there are teachers that are increasing non-cognitive, but not cognitive skills, surely there are schools that are doing the same. As a result, trying to assess if a school is “good” or “bad” relies on a complex web of preferences and objective measures that, quite frankly, cannot be taken into account in a centralized accountability system. We need something more sophisticated, and something that can respect a diverse conception of what “good” and “bad” means.

This is not to say that there should be no standards or accountability. The question is who imposes the accountability on whom. As I’ve noted previously, the absence of a government-imposed standard does not imply the lack of any standards. Rather, it leaves space for competing standards, which in turn fosters innovation and diversity. Parents can then evaluate the quality of education providers based on their own experience and the expert evaluations of appropriate external providers, and the entire system evolves as parents select the providers that best meet their children’s needs.

So yes, policymakers should be humble about what they know or think they know, but we can have greater confidence in a system that channels dispersed knowledge to produce greater quality and innovation. This is more than mere “intellectual play.” It’s the process by which we’ve seen enormous gains in productivity and quality in nearly every other sector in the last century — not top-down technocratic tinkering but bottom-up experimentation, evaluation, and evolution in a free market.


Texas Pummeling of California Begins to Resemble Child Abuse

May 8, 2014

(Guest Post by Matthew Ladner)

So you’ve probably all heard that Toyota is moving their HQ from California to Texas (their moving trucks will be driving right through **ahem** Arizona and New Mexico btw) but that is old news.  Now comes word that Houston has outdone the entire state of California for new housing starts.

 

 


Brilliant Health Care Solution Discovered!

April 3, 2014

20140403-175448.jpg

(Guest post by Greg Forster)

Shorter Yuval Levin: The Feds couldn’t make the Obamacare website in three years, then they outsourced the job to the private sector, which did it in six weeks, and are now bragging this proves Big Government can do the big jobs after all. So the obvious thing to do is outsource the actual provision of the health care to the private sector, declare victory and go home. The Gordian Knot of Big Government, cut at last!


Refuting Rauch and EPI on the Economics of Productivity

December 12, 2012

Hard Work U 3

(Guest post by Greg Forster)

Many readers of JPGB will be familiar with the hard-left, union-friendly Economic Policy Institute. A recent article by Jonthan Rauch uses some EPI graphs to argue that the U.S. economy no longer rewards working-class employees for productivity. Over on Hang Together, I say the graphs are deceptive. The problem is a decline in productivity in the workers, caused by – JPGB readers will be shocked – lousy K-12 schools (and also a loss of the older religious work ethic).

If you’re familiar with EPI’s work, you won’t be surprised – Jay, Marcus and I took on some very shoddy work they did on teacher pay back in Education Myths.


Jordan Increased Income Inequality on the Bulls while Making All the Players Wealthier

July 11, 2012

(Guest Post by Matthew Ladner)

Fun piece by Matthew Schonfield in the Journal today. Strangely the guys riding the pine on the Bulls in 1998 making four times as much as their equivalents in 1984 did not feel the need to bang on drums to protest income inequality. Also read Iowahawk’s particle physics/health care mashup.


Excellence in Failure

October 18, 2011

It’s long been a goal of mine to be so awful at a job that they have to pay me to leave.  Unfortunately, excellence in failure is something that has escaped me — even though I normally seek excellence in all things.

But you’ll be happy to know that football coaches, superintendents, CEOs, and even large corporations have all too often perfected the art of sucking so bad that people pay them for that failure.

These large severance packages are generally a problem when the people offering the package are doing so with OPM — Other People’s Money.  Athletic directors, school boards, boards of directors, and the government find it so much easier to be generous when the money they are offering to their failed coaches, superintendents, CEOs, and large corporations is not their own money.

The Occupy Wall Street folks have (rightly) highlighted the sweet deals offered to failed CEOs and corporations from tax funds, but they tend not to mention the frequency with which superintendents are given large severance packages with OPM.  Yes, the average size of the superintendent packages tends to be much smaller, but there are so many more of them that it adds up to real money.

A Chicago Tribune analysis last summer found:

The newspaper’s review of more than 100 superintendent contracts, financial records and severance agreements over a decade revealed that boards have handed out six-figure separation checks; district-paid health care; cash or retirement credit for hundreds of sick days; and, in one case, a Mercedes — all to be rid of superintendents….

“Boards have not been held responsible because they do not care about taxpayers, period. … They do not care about how much money they spend,” said Kenneth Williams, board president in Thornton Township High School District 205, which recently approved a $350,000-plus severance package for J. Kamala Buckner, a veteran superintendent. Williams tried unsuccessfully to rescind the package in May….

Using available state data, the Tribune tracked the number of superintendents in Illinois school districts from 2000 to 2010, finding nearly half had gone through three or more superintendents.  That turnover not only fuels buyout deals but can take a toll on issues ranging from policy to student achievement….

Changeover also means some superintendents get multiple severance payouts.

Rosemary Hendricks got a $132,000 settlement in 2009 after filing a lawsuit against Hoover-Schrum District 157, where she had served as superintendent about a year. Earlier, she had gotten a $75,000 settlement in Bellwood District 88, where she had a short stint as superintendent, records show. She is back as Bellwood superintendent.

Of course, sometimes these generous severance packages are the fault of boards, not the departing executive.  Boards sometimes choose to get rid of someone on a whim or simply because the majority composition of the board changes.  These reckless changes by boards to get rid of someone under contract or who could justly sue for wrongful termination, force boards to fork over large sums of OPM to avoid litigation.

My point is the irresponsibility that OPM encourages.  OPM encourages organizations to write contracts that are excessively long and generous and then require large severance packages to get out of them.  OPM encourages changing leadership without cause and at great cost.  OPM makes it so much easier to justify the bailout to avoid “systemic risk” or to promote “stability.”  And of course OPM facilitates coaches, superintendents, CEOs, and corporations to make unreasonable demands, take unreasonable risks, and fail catastrophically.

OPM encourages excellence in failure.


Follow

Get every new post delivered to your Inbox.

Join 3,229 other followers