Regulating School Choice: The Debate Continues

March 9, 2016

Design vs Experience

(Guest Post by Jason Bedrick)

Last week, the Cato Institute held a policy forum on school choice regulations (video here). Two of our panelists, Dr. Patrick Wolf and Dr. Douglas Harris, were part of a team that authored one of the recent studies finding that Louisiana’s voucher program had a negative impact on participating students’ test scores. Why that was the case – especially given the nearly unanimously positive previous findings – was the main topic of our discussion. Wolf and I argued that there is reason to believe that the voucher program’s regulations might have played a role in causing the negative results, while Harris and Michael Petrilli of the Fordham Institute pointed to other factors.

The debate continued after the forum, including a blog post in which Harris raises four “problems” with my arguments. I respond to his criticisms below.

The Infamous Education Productivity Chart

Problem #1: Trying to discredit traditional public schools by placing test score trends and expenditure changes on one graph. These graphs have been floating around for years. They purport to show that spending has increased much faster than expenditures [sic], but it’s obvious that these comparisons make no sense. The two things are on different scales. Bedrick tried to solve this problem by putting everything in percentage terms, but this only gives the appearance of a common scale, not the reality. You simply can’t talk about test scores in terms of percentage changes.

The more reasonable question is this: Have we gotten as much from this spending as we could have? This one we can actually answer and I think libertarians and I would probably agree: No, we could be doing much better than we are with current spending. But let’s be clear about what we can and cannot say with these data.

Harris offers a reasonable objection to the late, great Andrew Coulson’s infamous chart (shown below). Coulson already addressed critics of his chart at length, but Harris is correct that the test scores and expenditures do not really have a common scale. That said, the most important test of a visual representation of data is whether the story it tells is accurate. In this case, it is, as even Harris seems to agree. Adjusted for inflation, spending per pupil in public schools has nearly tripled in the last four decades while the performance of 17-year-olds on the NAEP has been flat.

U.S. Education Spending and Productivity

Producing a similar chart with data from the scores of younger students on the NAEP would be misleading because the scale would mask their improvement. But for 17-year-olds, whose performance has been flat on the NAEP and the SAT, the story the chart tells is accurate.

Voucher Regulations Are Keeping Private Schools Away

Problem #2: Repeating arguments that have already been refuted. Bedrick’s presentation repeated arguments about the Louisiana voucher case that I already refuted in a prior post. Neither the NBER study nor the survey by Pat Wolf and his colleagues provide compelling evidence that existing regulations are driving out potentially more effective private schools in the Louisiana voucher program, which was a big focus of the panel.

Here Harris attacks a claim I did not make. He is correct that there is no compelling evidence that regulations are driving out higher-quality private schools, but no one claimed that there was. Rather, I have repeatedly argued that the evidence was “suggestive but not conclusive” and speculated in my presentation that “if the enrollment trends are a rough proxy [for quality], though we can’t prove this, then it would suggest that the higher-quality schools chose not to participate” while lower-quality schools did.

Moreover, what Harris claims he refuted he actually merely disputed – and not very persuasively. In the previous post he mentions, he minimized the role that regulation played in driving away private schools:

As I wrote previouslythe study he cites, by Patrick Wolf and colleagues, actually says that what private schools nationally most want changed is the voucher’s dollar value. In Louisiana, the authors reported that “the top concern was possible future regulations, followed by concerns about the amount of paperwork and reports. When asked about their concerns relating to student testing requirements, a number of school leaders expressed a strong preference for nationally normed tests” (italics added). These quotes give a very different impression that [sic] Bedrick states. The supposedly burdensome current regulations seem like less of a concern than funding levels and future additional regulations–and no voucher policy can ever insure against future changes in policy.

Actually, the results give a very different impression than Harris states. The quote Harris cites from the report is regarding the concerns of participating schools, but the question at hand is why the nonparticipating schools opted out of the voucher program. Future regulations was still the top concern for nonparticipating schools, but current regulations were also major concerns. Indeed, the study found that 9 of the 11 concerns that a majority of nonparticipating private schools said played a role their decision not to participate in the voucher program related to current regulations, particularly around admissions and the state test.

"Views from Private Schools," by Brian Kisida, Patrick J. Wolf, and Evan Rhinesmith, American Enterprise Institute

Source: ”Views from Private Schools,” by Brian Kisida, Patrick J. Wolf, and Evan Rhinesmith, American Enterprise Institute (page 19)

Nearly all of the nonparticipating schools’ top concerns related to the voucher program’s ban on private schools using their own admissions criteria (concerns 2, 3, 5, 7, 8 and 11) or requiring schools to administer the state test (concerns 6, 9, 10, and possibly 7 again). It is clear that these regulations played a significant role in keeping private schools away from the voucher program. The open question is whether the regulations were more likely to drive away higher-quality private schools. I explained why that might be the case, but I have never once claimed that we know it is the case.

Market vs. Government Regulations in Education

Problem #3: Saying that unregulated free markets are good in education because they have been shown to work in other non-education markets. […] For example, the education market suffers from perhaps the worst information problem of any market–many complex hard-to-measure outcomes most of which consumers (parents) cannot directly observe even after they’ve chosen a school for their child. Also, since students can realistically only attend schools near their homes, and there are economies of scale in running schools, that means there will generally be few practical options (unless you happen to live in a large city with great public transportation–very rare in the U.S.). And the transaction costs are very high to switch schools. And there are equity considerations. And … I could go on.

Harris claims that a free market in education wouldn’t work because education is uniquely different from other markets. However, the challenges he lists – information asymmetry, difficulty measuring intangible outcomes, difficulties providing options in rural areas, transaction costs for switching schools – aren’t unique to K-12 education at all. Moreover, there is no such thing as an “unregulated” free market because market forces regulate. As I describe below, while not perfect, these market forces are better suited than the government to address the challenges Harris raises.

Information asymmetry and hard-to-measure/intangible outcomes:

Parents need information in order to select quality education providers for their children. But are government regulations necessary to provide that information? Harris has provided zero evidence that it is, but there is much evidence to the contrary. Here the disparity between K-12 and higher education is instructive. Compared to K-12, colleges and universities operate in a relatively free market. Certainly, there are massive public subsidies, but they are mostly attached to students, and colleges have maintained meaningful independence. Even Pell vouchers do not require colleges to administer particular tests or set a single standard that all colleges must follow.

So how do families determine if a college is a good fit or not? There are three primary mechanisms they use: expert reviews, user reviews, and private certification.

The first category includes the numerous organizations that rate colleges, including U.S. News & World Report, the Princeton Review, Forbes, the Economist, and numerous others like them. These are similar to sorts of expert reviews, like Consumer Reports, that consumers regularly consult when buying cars, computers, electronics, or even hiring lawyers – all industries where the non-expert consumer faces a significant information asymmetry problem.

The second category includes the dozens of websites that allow current students and alumni to rate and review their schools. These are similar to Yelp, Amazon.com, Urban Spoon and numerous other platforms for end-users to describe their personal experience with a given product or service.

Finally, there are numerous national and regional accreditation agencies that certify that colleges meet a certain standard, similar to Underwriters Laboratories for consumer goods. This last category used to be private and voluntary, although now it is de factomandatory because accreditation is needed to get access to federal funds.

None of these are perfect, but then again, neither are government regulations. Moreover, the market-based regulators have at least four major advantages over the government. First, they provide more comprehensive information about all those hard-to-measure and intangible outcomes that Harris was concerned about. State regulators tend to measure only narrow and more objective outcomes, like standardized test scores in math and English or graduation rates. By contrast, the expert and user reviews consider return-on-investment, campus life, how much time students spend studying, teaching quality, professor accessibility, career services assistance, financial aid, science lab facilities, study abroad options, and much more.

Second, the diversity of options means parents and students can better identify the best fit for them. As Malcolm Gladwell observed, different people give different weights to different criteria. A family’s preferences might align better with the Forbes rankings than the U.S. News rankings, for example. Alternatively, perhaps no single expert reviewer captures a particular family’s preferences, in which case they’re still better off consulting several different reviews and then coming to their own conclusion. A single government-imposed standard would only make sense if there was a single best way to provide (or at least measure) education, we knew what it was, and there was a high degree of certainty that the government would actually implement it well. However, that is not the case.

Third, a plethora of private certifiers and expert and user reviews are less likely to create systemic perverse incentives than a single, government standard. As it is, the hegemony of U.S. News & World Report’s rankings created perverse incentives for colleges to focus on inputs rather than outputs, monkey around with class sizes, send applications to students who didn’t qualify to increase their “selectivity” rating, etc. If the government imposed a single standard and then rewarded or punished schools based on their performance according to that standard, the perverse incentives would be exponentially worse. The solution here is more competing standards, not a single standard.

Fourth, as Dr. Howard Baetjer Jr. describes in a recent edition of Cato Journal, whereas “government regulations have to be designed based on the limited, centralized knowledge of legislators and bureaucrats, the standards imposed by market forces are free to evolve through a constant process of evaluation and adjustment based on the dispersed knowledge, values, and judgment of everyone operating in the marketplace.” As Baetjer describes, the incentives to provide superior standards are better aligned in the market than for the government:

Incentives and accountability also play a central role in the superiority of regulation by market forces. First, government regulatory agencies face no competition from alternative suppliers of quality and safety assurance, because the regulated have no right of exit from government regulation: they cannot choose a better supplier of regulation, even if they want to. Second, government regulators are paid out of tax revenue, so their budget, job security, and status have little to do with the quality of the “service” they provide. Third, the public can only hold regulators to account indirectly, via the votes they cast in legislative elections, and such accountability is so distant as to be almost entirely ineffectual. These factors add up to a very weak set of incentives for government regulators to do a good job. Where market forces regulate, by contrast, both goods and service providers and quality-assurance enterprises must continuously prove their value to consumers if they are to be successful. In this way, regulation by market forces is itself regulated by market forces; it is spontaneously self-improving, without the need for a central, organizing authority.

In K-12, there are many fewer private certifiers, expert reviewers, or websites for user reviews, despite a significantly larger number of students and schools. Why? Well, first of all, the vast majority of students attend their assigned district school. To the extent that those schools’ outcomes are measured, it’s by the state. In other words, the government is crowding out private regulators. Even still, there is a small but growing number of organizations like GreatSchools, Private School Review, School Digger, andNiche that are providing parents with the information they desire.

Options in rural areas:

First, it should be noted that, as James Tooley has amply documented, private schools regularly operate – and outperform their government-run counterparts – even in the most remote and impoverished areas in the world, including those areas that lack basic sanitation or electricity, let alone public transportation. (For that matter, even the numerous urban slums where Tooley found a plethora of private schools for the poor lack the “great public transportation” that Harris claims is necessary for a vibrant education market.) Moreover, to the extent rural areas do, indeed, present challenges to providing education, such challenges are far from unique. Providers of other goods and services also must contend with reduced economies of scale, transportation issues, etc.

That said, innovations in communication and transportation mean these obstacles are less difficult to overcome than ever before. Blended learning and course access are already expanding educational opportunities for students in rural areas, and the rise of “tiny schools” and emerging ride-sharing operations like Shuddle (“Uber for kids”) may soon expand those opportunities even further. These innovations are more likely to be adopted in a free-market system than a highly government-regulated one.

Test Scores Matter But Parents Should Decide 

Problem #4: Using all this evidence in support of the free market argument, but then concluding that the evidence is irrelevant. For libertarians, free market economics is mainly a matter of philosophy. They believe individuals should be free to make choices almost regardless of the consequences. In that case, it’s true, as Bedrick acknowledged, that the evidence is irrelevant. But in that case, you can’t then proceed to argue that we should avoid regulation because it hasn’t worked in other sectors, especially when those sectors have greater prospects for free market benefits (see problem #3 above). And it’s not clear why we should spend a whole panel talking about evidence if, in the end, you are going to conclude that the evidence doesn’t matter.

Once again, Harris misconstrues what I actually said. In response to a question from Petrilli regarding whether I would support “kicking schools out of the [voucher] program” if they performed badly on the state test, I answered:

No, because I don’t think it’s a wise move to eliminate a school that parents chose, which may be their least bad option. We don’t know why a parent chose that school. Maybe their kid was being bullied at their local public school. Maybe their local public school that they were assigned to was not as good. Maybe there was a crime problem or a drug problem.

We’re never going to have a perfect system. Libertarians are not under the illusion that all private schools are good and all public schools are bad… Given the fact that we’ll never have a perfect system, what sort of mechanism is more likely to produce a wide diversity of options, and foster quality and innovation? We believe that the market – free choice among parents and schools having the ability to operate as they see best – has proven over and over again in a variety of industries to have better outcomes than Mike Petrilli sitting in an office deciding what quality is… as opposed to what individual parents think [quality] is.

Harris then responded by claiming that I was saying the evidence was “irrelevant,” to which I replied:

It’s irrelevent in terms of how we should design the policy, in terms of whether we should kick [schools] out or not, but I think it’s very important that we know how well these programs are working. Test scores do measure something. They are important. They’re not everything, but I think they’re a pretty decent proxy for quality…

In other words, yes, test scores matter. But they are far from the only things that matter. Test scores should be one of many factors that inform parents so that they can make the final decision about what’s best for their children, rather than having the government eliminate what might well be their least bad option based on a single performance measure.

I am grateful that Dr. Harris took the time both to attend our policy forum and to continue the debate on his blog afterward. I look forward to continued dialogue regarding our shared goal of expanding educational opportunity for all children.

(Cross-posted at Cato-at-Liberty.)

 


Overregulation Is All You Need: A Response to Paul Bruno

February 29, 2016

6-blind-men-hans

(Guest Post by Jason Bedrick)

Over at the Brookings Institution’s education blog, Paul Bruno offers a thoughtful critique of  Overregulation Theory (OT), the idea that government regulations on school choice programs can undermine their positive effects. Bruno argues that although OT is “one of the most plausible explanations” of the negative results that two studies of Louisiana’s voucher program recently found, it is not “entirely consistent with the available evidence” and “does not by itself explain substantial negative effects from vouchers.”

I agree with Bruno–and have stated repeatedly–that the studies’ findings do not conclusively prove OT. That said, I believe both that OT is consistent with the available evidence and that it could explain the substantial negative effects (though I think it’s likely there are other factors at play as well). I’ll explain why below, but first, a shameless plug:

On Friday, March 4th at noon, the Cato Institute will be hosting a debate over the impact of regulations on school choice programs featuring Patrick Wolf, Douglas Harris, Michael Petrilli, and yours truly, moderated by Neal McCluskey. If you’re in the D.C. area, please RSVP at this link and join us! Come for the policy discussion, stay for the sponsored lunch!

Is the evidence consistent with Overregulation Theory?

Bruno notes that the differences in enrollment trends between participating and non-participating private schools is consistent with OT. Participating schools had been experiencing declining enrollment in the decade before the voucher program was enacted whereas non-participating schools had slightly increasing enrollment on average. This is consistent with the OT’s prediction that better schools (which were able to maintain their enrollment or grow) would be more likely eschew the vouchers due to the significant regulatory burden, while the lower-performing schools (which were losing students) were more desperate for students and funding, and were therefore more willing to jump through the voucher program’s regulatory hoops. However, Bruno calls this evidence into question:

For one thing, the authors of the Louisiana study specifically check to see if learning outcomes vary significantly between schools experiencing greater or lesser prior enrollment declines, and find that they do not. (Bedrick acknowledges this, but doubts there was enough variation in the enrollment trends of participating schools to identify differences.)

We should be skeptical of the explanatory value of the study’s enrollment check. There is no good reason to assume that the correlation between enrollment growth or decline among the small sample of participating schools (which had significantly negative growth, on average) is the same as among all private schools in the state. Making such an assumption is like a blind man holding onto the truck of an elephant and assuming that he’s holding a snake.

The study does not show the variation in enrollment trends among the participating and non-participating schools, but we could imagine a scenario where the enrollment trend among participating schools ranged, say, from -25% to +5% while the range at non-participating schools was -5% to +25%. As shown in the following charts (which use hypothetical data), there may be a strong correlation between enrollment trends and outcomes among the entire population, while there is little correlation in the subset of participating schools.

Enrollment Growth and Performance, Participating Private Schools (Hypothetical)

Screen Shot 2016-02-29 at 4.57.05 PM

Enrollment Growth and Performance, All Private Schools (Hypothetical)

Screen Shot 2016-02-29 at 10.05.29 PM.png

In short, looking at the relationship between enrollment growth and performance in the narrow subset of participating schools doesn’t necessarily tell us anything about the relationship between enrollment growth and performance generally. Hence the study’s “check” that Bruno cites does not provide evidence against OT.

Is there evidence that regulations improve performance?

Bruno also cites evidence that regulations can have a positive impact on student outcomes:

Joshua Cowen of Michigan State University also points out that there is previous evidence of positive effects from accountability rules on voucher program outcomes in other states (though regulations may differ in Louisiana).

The Cowen article considers the impact of high-stakes testing imposed on the Milwaukee voucher program during a multi-year study of that program. The “results indicate substantial growth for voucher students in the first high-stakes testing year, particularly in mathematics, and for students with higher levels of earlier academic achievement.” But is this strong evidence that regulations improve performance? One of the authors of both the original Milwaukee study and the cited article–JPG all-star, Patrick Wolf–cautions against over-interpreting these results:

Ours is one study of what happened in one year for one school choice program that switched from low-stakes testing to high-stakes testing.  As we point out in the report, it is entirely possible that the surge in the test scores of the voucher students was a “one-off” due to a greater focus of the voucher schools on test preparation and test-taking strategies that year.  In other words, by taking the standardized testing seriously in that final year, the schools simply may have produced a truer measure of student’s actual (better) performance all along, not necessarily a signal that they actually learned a lot more in the one year under the new accountability regime.

If we had had another year to examine the trend in scores in our study we might have been able to tease out a possible test-prep bump from an effect of actually higher rates of learning due to accountability.  Our research mandate ended in 2010-11, sadly, and we had to leave it there – a finding that is enticing and suggestive but hardly conclusive.

It’s certainly possible that the high-stakes test improved actual learning. But it’s also possible–and, I would argue, more probable–that changing the stakes just meant that the schools responded to the new incentive by focusing more on test-taking strategies to boost their scores.

For that matter, even if it were true that the regulations actually improved student learning, that does not contradict Overregulation Theory. Both advocates and skeptics of the regulations believe that schools respond to incentives. Those of us who are concerned about the impact of the regulations don’t believe that they can’t improve performance. Rather, our concern is that regulations imposed from above are less effective at improving performance than the incentives created by direct accountability to parents in a robust market in education, and may have adverse unintended consequences.

To explain: We’re concerned that regulations forbidding the use of a school’s preferred admissions standards or requiring the state test (which is aligned to the state curriculum) might drive away better-performing schools, leaving parents to choose only among the lower-performing schools. We’re concerned that price controls will inhibit growth, providing schools with an incentive only to fill empty seats rather than to scale up. We’re concerned that mandatory state tests will inhibit innovation and induce conformity. None of these concerns rule out the possibility (or, indeed, the likelihood) that over time, requiring private schools to administer the state test and report the results and/or face sanctions based on test performance will improve the participating schools’ performance on that test.

Again: we agree that schools respond to incentives. We just think the results of top-down incentives are likely to be inferior to the results of bottom-up choice and competition, which have proved to be powerful tools in so many other fields for spurring innovation and improving quality.

Can Overregulation Theory alone explain the negative results in Louisiana?

Finally, Bruno questions whether OT alone explains the Louisiana results:

[E]ven if regulation prevented all but the worst private schools from participating, this would explain why students did not benefit from transferring into them, but not why students would transfer into them in the first place.

So Overregulation Theory might be part of the story in explaining negative voucher effects in Louisiana, but it is not by itself sufficient. To explain the results we see in the study, it is necessary to tell an additional story about why families would sort into these apparently inferior schools.

Bruno offers a few possible stories–that parents select schools “that provide unobserved benefits,” that the voucher program “induced families to select inferior schools,” or that parents merely “assume any private school must be superior to their available public schools”–but any of these can be consistent with OT.  Indeed, the second story Bruno offers is practically an extension of OT: if the voucher regulations truncate supply so that it is dominated by low-quality schools, and the government gives false assurances that they have vetted those schools, then it is likely that we will see parents lured into choosing inferior schools.

That’s not to say that there are no other factors causing the negative results. It’s likely that there are. (I find Douglas Harris’s argument that the private schools’ curricula did not align with the state test in the first year particularly compelling, though I don’t think it entirely explains the magnitude of the negative results.) We just don’t have any compelling evidence that OT is wrong, and OT can suffice to explain the negative results.

I will conclude as I began: expressing agreement. I concur with Bruno’s assessment that “it is likely that the existing evidence will not allow us to fully adjudicate between competing hypotheses.” Indeed, it’s likely that future evidence won’t be conclusive either (it rarely is), but I hope that further research will shed more light on this important question. Bruno concludes by calling for greater efforts to “understand how families determine where their children will be educated,” noting that by understanding how and why parents might make “sub-optimal — or even harmful” decisions will help “maximize the benefits of school choice while mitigating its risks.” These are noble goals and I share Bruno’s desire to pursue them. I just hope that policymakers will approach what we learn with a spirit of humility about what they can accomplish.


Gentrification is the primary driver of District of Columbia Academic Gains

February 9, 2016

The Capital is playing games with some limited success but low-income kids still hunger for academic gains.

(Guest Post by Matthew Ladner)

New study out today from the Heritage Foundation authored by yours truly on education in the District of Columbia. I will serialize the study a bit with a chart or two per post. The view I had of K-12 in the nation’s capital going into the project did not survive my investigation. My initial view going into the project is summarized in this chart:

 

Ladner DC 4

Moving your 8th grade math scores from 230 in 1990 to 263 (combined district and charter) in 2015 is a lot of progress- by far the largest gains in the nation. While things are somewhat worse if we look at District of Columbia Public Schools (DCPS) scores alone (258) they still are far above the earliest measure. Better yet- when you examine charter scores in isolation from DCPS scores, the scores are higher still. Any actual education going on in DCPS back in 1990 looks to have been mostly accidental- I’m not sure what you would score on the NAEP math test if you just answered “B” every time, but with 3% of DC students scoring proficient and 83% scored “Below Basic” on the NAEP 8th grade math exam in 1990, it could not have been much lower. In 2015 19% of DC students scored proficient and only 49% scored Below Basic. The improvement is undeniable: time to CeleNAEP? Those gains dwarf anything seen out in the states, and the charter school sector getting close to half of the students is clearly a major driver of improvement. Twirl for me, girl on fire!

Alas while this story is true it is far from complete. What is going on in DC is both complex, partially encouraging and in the end very disturbing.

Overall enrollment in DC (district and charter) had been growing in recent years along with average incomes. A complex phenomenon is underway in which sophisticated young parents have figured out that they don’t need to move to Maryland or Virginia if they can find a spot in the right district or charter school in the District. In the end your kid doesn’t get educated by a district or a CMO but rather by a school. DC is the champion for NAEP gains, but it is also the champion for achievement gaps.

You can see the gentrification going on both in DC’s statistics and with the naked eye walking around town. Somebody keeps buying those million plus dollar brown stones and some of them are parents. This begs the question: how much of DC’s apparent academic gains owes to gentrification?

Sadly the answer is- affluent children have banked the vast majority of DC NAEP gains over the last decade.

Ladner DC 7

There is a lot going on in this chart so pay close attention. These are again NAEP 8th grade math gains by family income (FRL status) over the last decade. First look at the light blue columns- gains for FRL eligible kids. DCPS district equals an 8 point gain, which is indistinguishable from the national average of 7 points. The 17 point gain for free and reduced lunch kids attending DC charter schools is the only real bright spot for disadvantaged kids in the public school system despite decades of reform. We used 8th grade math for purposes of illustration but you see a similar pattern across the NAEP exams.

Now observe the dark blue columns- DC kids whose incomes are too high for a Free or Reduced price lunch under federal guidelines. Here the gains are truly extraordinary- a 28 point gain for non-FRL kids attending charter schools and a 39 point gain for middle to high income district students. I’m placing my bet now that this isn’t solely due to schools in Georgetown doing an ever-better job educating kids with law-firm names, but also to the fact that people who once fled to Va and MD finding a spot that suited them in DC.

So in 1990 let’s estimate that the number of DCPS FRL kids scoring proficient on 8th grade math as effectively zero. Twenty five years later, that figure is up to 8 percent for district kids (charters excluded). DCPS in other words remains largely what it has always been- an organization far better at employing adults than meeting the needs of disadvantaged children. As we will see in a future post, the academic results of DCPS continue to disappoint even in comparisons against other urban districts despite 15 years of strong progress and gentrification.

I am Ozymandias-Queen of Queens! Look upon the ineffectiveness of my broom ye mighty and despair!

Overall the situation in DC K-12 is very complex- with both positive trends and heartbreaking stagnation. Regardless of where you are coming from on the political/philosophical side of things, if you are a DC taxpayer you should not stand for this state of affairs as it touches upon economically disadvantaged children. The above chart shows is that despite a truly shocking amount of tax effort and a decade and a half of reform, what DCPS has figured out how to do is to give the most academically to the kids born on third base. Mind you this is much better than giving approximately nothing to anyone a la DCPS circa 1990, but that is in the big picture a cold comfort. In the end it is very positive for the fiscal health of the District of Columbia that third base parents can in fact get a quality education for their children it in the right bits of DCPS. Moreover, those third base parents are paying a mind-numbing level of tax and they deserve a quality education for their children.

So does everyone else.

 


Petrilli’s Regulatory Porridge

January 28, 2016

goldilocks

(Guest Post by Jason Bedrick)

Fordham’s Michael Petrilli offers new taxonomy for school choice tribes dividing the school choice world in three: Purists, Nannies, and Realists.

First, as Matt noted, this is not Mike’s first foray into Hemisphere Fallacy territory. (Or the second. Or even the third.) Like the guy who thinks anyone who is more religious is crazy and anyone who is less religious is a heretic, Mike thinks he has found the Perfect Goldilocksian Mean and everyone else is wrong. In Mike’s view, those who support more regulation than he does are paternalist Nannies, and those who support less regulation are utopian Purists, but the temperature of his regulatory porridge is just right.

Second, as I noted on Twitter, it’s adorable that Mike thinks he’s not a Nanny. He decries their “micromanagement” but he supports forcing private schools to administer the state test (de facto determining what is taught when and even how), as well as price controls that economists will tell you leads to shortages and obliterates the essential price signal (without which we may have competition, but we most certainly do not have a functioning market). He may fancy himself a “Realist” but, if these categories really mean anything, he just has minor disagreements with his fellow Nannies.

Third, Mike is engaging, once again, in the Means and Ends Fallacy. It goes something like this:

I think X is a problem. I believe Solution Y solves X. Group A opposes Solution Y, therefore Group A must not think X is really a problem.

Of course, this is a fallacy because it is entirely possible (as is the case here) that Group A agrees that X is a problem but doesn’t think Solution Y actually solves it. Mike thinks his preferred regulations solve the problem of bad schools, but we think those regulations are more likely to have adverse effects. More on that in a moment.

Mike accuses the “Purists” (those who, like Milton Friedman, conclude based on the evidence from nearly every other industry that markets spur innovation and lead to greater quality and efficiency) of being utopian. He writes:

Start with the Purists. I’m skeptical of all utopian visions, including theirs—one imagining that a full-fledged system of choice (perhaps through universal Education Savings Accounts) will yield greater innovation, productivity, and customer satisfaction—and produce better-educated young people to boot.

But there’s nothing utopian about that. We see that ESAs have already begun to produce greater innovation, productivity, and customer satisfaction. We don’t yet have any data on test scores or graduation rates, but we have no reason to believe that ESAs will underperform the many voucher programs that have produced positive results. No one in the free-market crowd is expecting miracles. We’re expecting the sort of incremental improvement that the market regularly brings about through the process of experimentation, evaluation, and evolution.

(For that matter, the most utopian schemes in education come from the Nannies. Can you imagine a more fantastically utopian scheme than “No Child Left Behind”? Is there a more utopian slogan than that anywhere in education policy? And did anyone in the administration really believe we’d ever achieve 100% proficiency in any state, let alone every state? Either they set up the nation to fail or they were delusional. Or perhaps they just set up the DOE for a naked power grab. But I digress.)

Mike’s central challenge to the Friedmanite crowd is the Payday Lender Problem. What do we do about bad private schools?

First of all, Mike doesn’t have a whole lot of evidence that the government does a better job ensuring quality than the market. Indeed, the Louisiana debacle should give him great pause about that article of faith.

Second, eliminating the least-bad option doesn’t guarantee a better option. Payday lending serves an important function in the market (in the Third World, we call it “microlending,” a concept for which Muhammad Yunus won a Nobel Peace Prize). Poor people who need funds to cover rent or buy food while waiting for payday often turn to payday lenders. If they repay the loan on time, the fees are generally marginal. If they repay late, the interest rates can be exorbitant, especially if (misleadingly) expressed in annual terms. (The interest is so high both because the loans are so small and because the rate of default is so high, which is why banks generally just refuse to lend to the poor.) But eliminating the payday lenders can have serious unintended consequences that make the poor even worse off. The payday lender may charge a steep fee for late payment, but at least Rocky Balboa doesn’t come break your legs.

Kicking a school with poor test scores out of a voucher program doesn’t guarantee those poor kids a seat at a better school. Rather, the state just eliminates that kid’s least-bad alternative. Even in Louisiana, where the voucher schools appear to be doing much worse on the state test than the district school alternatives, the families who chose those schools may well have had good reasons for doing so. Perhaps they were safer. Perhaps they had higher graduation rates. We don’t know. But those families chose them for a reason and they may well be worse off overall if deprived of that choice.

Third, as Michael McShane explained previously, the market process has proven time and again to significantly improve absolute quality (and efficiency) over time:

Cars today are uniformly better than cars in 1950. They are safer. They are faster. They are more comfortable. They are more fuel efficient.  But it wasn’t a clear upward-sloping line to get here. People bought Edsel’s in the 50’s, Corvairs in the 60’s, Chevettes in the 70’s, Yugo’s in the 80’s, Suzuki Sidekicks in the 90’s, and Pontiac Aztecs in the 00’s. These were bad cars.

But “bad” has two meanings in this case, an objective one and a relative one. There are relatively bad cars out there today. That is, my hail-damaged ’05 Kia Spectra with no cruise control and a blown-out right front speaker is worse than Jay-Z’s Maybach on almost every calculable measure, relatively speaking. But my Spectra, which is still purring like a kitten after over 100,000 miles with darn near nothing more than oil changes, tires, and brake pads is a helluva lot better than the burn-out-after-five-years cars that automakers made for decades.  That’s absolute quality.

Markets work when the spectrum of relative quality drives improvements in absolute quality.  Someone sees my little tin can driving down the road and says “I want to buy a car that doesn’t look like it’s going to blow away in a stiff breeze” and cars get less tin-canny.  Someone buys a Ford Excursion and then gas prices go up and says, “I’m never doing that again” and cars get more fuel efficient. It’s a slow winnowing process, but over time it is superior to centralized systems, that, for example, made the Trabant in an essentially unchanged manner for over three decades.

Rather than thinking we can regulate bad schools out of existence, a better goal is to develop a system that continuously improves what we think a “bad” school is.

Mike Petrilli is right to be worried about kids who are in bad schools today, but the regulations he proposes to ensure that those students are attending relatively good schools interfere with the market process that could otherwise be driving up absolute quality for everyone (and, as Louisiana has shown, those kids may end up in low-performing schools anyway).

Imagine if government officials, following Mike’s logic, had decided decades ago that every low-income family should have access to a phone. Now, these Realist officials aren’t Nannies — they’re not going to have the government make the phones or micromanage the specs. They’re just going to ensure that everyone has access to a good phone, so they create a phone voucher but prohibit companies selling phones from charging more than the value of the voucher. What would have happened?

Well, there’s the seen and the unseen. We would have seen, perhaps, that everyone would have had access to a phone and many would have applauded that (although given the price controls, it’s likely that supply would not have met demand). But what we wouldn’t have seen was that the iPhone had not been invented. With no way to charge more than the meager voucher, there’d be no market for expensive smartphones. And that wouldn’t just have harmed the wealthy, it would also have harmed the poor. After all, Walmart now sells a $10 smartphone that has better specs that the original iPhone. Innovations that at first benefit the wealthier early adopters tend to benefit even the poor after a while.

In short, Mike’s admirable passion to help the poor immediately through state action may well harm them in the medium-to-long run without any guarantee of actually helping them in the short run. That doesn’t sound very Realist to me.

 

 

 


The Choice Genie Continues to Flow out of the Florida Bottle

January 25, 2016

(Guest Post by Matthew Ladner)

RedefinED has published their sixth look at the changing landscape of Florida education, reporting over 88,000 new choice students over the last two years.

Florida choice


I Love It when a (Decentralized) Plan Comes Together…

January 14, 2016

(Guest Post by Matthew Ladner)

Jason and John White have been having a public dispute over the interpretation of the NBER Louisiana study. I await Jason’s response, and given the importance of the subject I hope both sides of this debate will take pains to avoid vitriol as it unfolds. A reasoned examination of the evidence will best serve us, and everyone should acknowledge in advance that we bring a variety of assumptions to the table with us in this debate.

There is one point in John’s response that I would like to address presently. Early on the Superintendent says:

More important, however, is the larger implication I take from Mr. Bedrick’s thesis: that private school choice advocates in America, Mr. Bedrick among them, have failed to establish a coherent, prevailing belief system about the role of private schools in providing an education of measured quality, at scale, for the nation’s most disadvantaged youth.

Towards the end he says:

Absent better systemic answers than those offered by ideologues, publicly funded private school choice for all children will continue to be more of a factor in legislative debates and scholarly conferences than in the homes and neighborhoods of America’s youth.

The use of the word “measured” in the first statement brings up the observer effect from physics: the act of observing can in itself change what you attempt to measure. People can have honest disagreements over how much private school participation we are willing to sacrifice for what type of measurement, but let’s not start with the premise that the answer is “whatever it takes no matter what.” For almost going on two decades now for instance the McKay Scholarship program has reigned as the largest voucher program in the country. Would I prefer having an evaluation component of some sort? Yes-I’m pretty confident that we would learn valuable information. Would I be willing to do it if a large portion of participating schools were likely to pass on participating? No- I would be content to let parents work out testing on an individual basis with their schools.

In other words I want to prioritize the needs of parents over my needs as an advocate.

On the issue of scale and having a plan, I think we could have a long debate about this, but ultimately I don’t think it is necessary. I think that the sort of effort that Superintendent White is referencing may relate to the sort of private philanthropic efforts we see going on in the New Orleans charter school space. An unexplored possibility to consider by the way, is that the Herculean private human capital efforts we see going on in the charter space may have eclipsed the private school sector. How many rational actors would open a private as opposed to a charter school in New Orleans given the totality of policy and private philanthropic efforts? But I digress…

The reason I don’t think we need to spend too much time debating the sort of genuinely heroic efforts we see going on in the New Orleans charter space: I don’t think we can afford to scale them. You don’t see it discussed often, but ah, well, the cost per teacher placement from sources like Teach for America has not been going down. Quite the opposite from what I understand. Don’t get me wrong- I love TFA kids and I fully support the efforts of philanthropists in rebuilding the New Orleans education system. I simply have my doubts that our collective philanthropic efforts could have handled a district the size of Houston had Katrina veered that way, much less the rest of the country.

This raises the question- can we achieve “an education of measured quality, at scale, for the nation’s most disadvantaged youth” without these sort of efforts? If you care about the disadvantaged, the answer had better be “yes” because otherwise they will be remain holding the short end of the stick in life. To answer the question, I decided to examine the NAEP results and trends for low-income kids in the relatively regulated and highly philanthropic supported charter sector in Louisiana with similar kids in the Wild West and (relatively) lightly supported Arizona charter sector.

To get as close to apples to apples as possible, I compared the results for Free and Reduced lunch eligible students in general education programs attending charter schools in both states. The general education focus is due to possible differences in participation among ELL and SPED students in the two states. Louisiana has a comprehensive effort to support charters and to shut them down when they under-perform. If you see race as a proxy for other unmeasured factors etc. the average low-income charter child in Arizona is likely to be Hispanic, whereas I’m guessing in Louisiana they are more likely to be Black. Every state has their story about how their poor kids are the toughest to educate in the world (here is AZ it is “they come up North and they don’t even read Spanish!”) but I view all such claims as suspect-poor kids are tough to educate everywhere.

In terms of charter sectors, Arizona has an almost entirely decentralized process of school creation, and the only people making plans are those opening schools and parents seeking places for their child- no common app, the state rarely shuts a school down. etc. We have some TFA kids but I met more of them in a visit to the French Quarter than in 13 years of living here in AZ. The West in short doesn’t get any wilder.

The NAEP will only provide data for this comparison between 2013 and 2015, and I make no claims regarding this being a definitive comparison, merely suggestive. So with those caveats: here is what the gains look like across all four NAEP exams for FRL general ed charter students:

AZLA

So both sectors did well in generating gains overall for low-income kids- so bully for them. Louisiana charters enjoyed larger gains in 4th grade gains, but Arizona charters enjoyed a still larger advantage at the 8th grade level. If we look at the overall scores for these students rather than the gains, we see no decisive advantage for either state.

AZLA2

Those who care about the interests of poor children should celebrate the ability of Arizona charter schools to deliver big gains for low-income kids. They did it without direction from the state and without massive support from philanthropists. This is to be celebrated in large part because we don’t have enough philanthropy to do New Orleans everywhere even if we wanted to.

So- as one of the school choice Jeffersonian types, I’ll address Superintendent White’s point about a plan squarely. What is my plan? On charter schools, my plan is to let ‘er rip. It’s working out a splendid fashion and I love it when a plan comes together…

On private choice, I’d like to see a system that gives a meaningfully higher level of public subsidy to disadvantaged children (low-income, ELL, SPED) with some light touch academic transparency measures (NNR testing required by students rather than schools, academic studies, parent surveys) and then let that rip too. The great virtue of this plan in my view is that it doesn’t require me or anyone else to manage or direct it.

In other words, I think school choice technocrats should aspire to be in control of as little as possible. The NBER study reinforces my view of its desirability.

UPDATE: Jason’s response to Superintendent White is up at Ed Next


The Folly of Overregulating School Choice: A Response to Critics

January 8, 2016

84c

[Guest Post by Jason Bedrick]

Earlier this week, NBER released the first random-assignment study ever to find a negative impact from a school voucher program. Previous gold standard studies had almost unanimously found modest positive effects from school choice, which raises the obvious question: what makes the Louisiana Scholarship Program (LSP) so different?

In an article for Education Next, I argued that, “although not conclusive, there is considerable evidence that the problem stemmed from poor program design.” The LSP is one of the most heavily regulated school choice programs in the nation, and that burden has led to a very low rate of private school participation.  Only about one-third of Louisiana private schools accept voucher students, a considerably lower rate than in most other states. From a survey of private school leaders conducted by Brian Kisida, Patrick J. Wolf, and Evan Rhinesmith for the American Enterprise Institute, we know that the primary reason private schools opted out of the voucher program was their concerns over the regulatory burden, particularly those regulations that threatened their character and identity. For example, voucher-accepting schools in Louisiana may not set their own admissions criteria, cannot charge families more than the value of the voucher (a meager $5,311 on average in 2012), and must administer the state test.

We also know from the NBER study that the participating and non-participating private schools differ in at least one important respect. Whereas the non-participating schools experienced modest growth over the decade before the voucher program was expanded statewide (about 3 percent, on average), the participating schools had been experiencing a significant decline in enrollment (about 13 percent, on average). In other words, schools that were able to attract students tended to reject the vouchers while voucher schools tended to be those where enrollment had been dropping.*

The difference in enrollment trends suggests that the LSP’s regulatory burden had the opposite of its intended effect: discouraging higher-performing schools from participating, leaving only the lower-performing schools that were so desperate to reverse their declining enrollment and increase their funding that they were willing to do whatever the voucher program required.

Several other researchers and education reform advocates reached similar conclusions, including Matthew Ladner, Adam Peshek, Michael McShane, Lindsey Burke, and Jonathan Butcher. However, others expressed skepticism about what I shall call the Overregulation Theory, and proposed alternative explanations for the LSP’s poor results.

Writing at Education Week, Douglas Harris of the Education Research Alliance for New Orleans concedes that “regulation probably does reduce the number of private schools, especially the number of higher-performing private schools,” but he still believes the Overregulation Theory is “premature.” Harris instead offers two potential alternatives: 1) the improved public/charter school performance in New Orleans made the performance of the private sector look relatively worse; and 2) the curriculum at most private schools may not have been aligned to the state test, so the poor performance merely reflects that lack of alignment rather than poor performance.

Harris’s first theory is explicitly rejected by the NBER study. On the third page of the study, the authors write: “Negative voucher effects are not explained by the quality of public fallback options for LSP applicants: achievement levels at public schools attended by students lotteried out of the program are below the Louisiana average and comparable to scores in low- performing districts like New Orleans.” In other words, the public school alternatives are not so great and the performance of the participating private schools is considerably worse.

That said, Harris’s second theory, which Jason Richwine also suggested, is plausible as a contributing factor. However, it is no more plausible than the Overregulation Theory. Indeed, whereas the differences in enrollment trends between voucher and non-voucher private schools provide some suggestive evidence for the Overregulation Theory, Harris provides no evidence to support the Nonaligned Test Theory. How many voucher schools were already aligned with the state curriculum and/or administered the state test? At this point, we do not know. Moreover, to the extent that testing nonalignment explains some of the very large 0.4 standard deviation difference in math scores, it is unlikely that it explains all or even most of that difference. Then again, Harris stated that he will be releasing the results of his own research on the LSP, so it’s likely he knows something that I do not.

Harris also notes that the NBER study only examined the results of one year of one program. He is certainly correct that we need more data over time to draw firmer conclusions, which is one reason I presented my interpretation as “not conclusive” and wrote that “the regulationsmay have had the opposite of their intended effect” (emphasis added). And, indeed, there is some evidence that voucher schools improved slightly in the third year since the statewide expansion (although if the voucher schools were their own district, they’d still be the fifth-worst of 76 in the state).

Nevertheless, such strongly negative results should give reform advocates great pause about the regulatory strategies employed in Louisiana. We know the regulatory burden chased away most private schools, and we have evidence that the voucher-accepting schools had been struggling with declining enrollment. If we want to better understand the LSP’s atypically disastrous performance, its program design is the logical place to start.

*UPDATE: Prof. Josh Goodman of Harvard University points out that the NBER study’s “estimates show consistent negative impacts irrespective of previous enrollment trends.” If there had been wide variation in enrollment trends among the participating schools, that would count as evidence against the Overregulation Theory. However, if the enrollment trends among participating schools were consistently negative, with the variation mostly limited to how fast the participating schools had been losing students, then it would still be consistent with the Overregulation Theory.

[Originally posted at Cato-at-Liberty.]