Jay P. Greene's Blog

Many of the most powerful backers of school choice are embracing a high-regulation approach.  Their interests have shifted from promoting choice as the goal to using choice as a mechanism for obtaining more quality schools.  They don’t trust that choice produces quality.  They want a fairly heavy dose of regulation to prevent bad schools from being included among the options available to families.  They want to control key aspects of school operations to prevent schools from becoming bad.  And they want a powerful regulator — a portfolio manager or harbor master — who will identify and remove bad schools from choice programs.

I think this approach is deeply flawed.  I understand that political reality requires some amount of reasonable regulation.  But the view that regulation, not choice itself, is the main driver of quality improvement is completely wrong.  My fear is that just when school choice is achieving escape velocity as a self-sustaining and expanding policy, the love for high-regulation may do serious harm to these programs and the children they intend to help.

Over a series of blog posts I intend to describe the arguments folks have for high-regulation and why I think those arguments are mistaken.  First let me describe the high-regulation approach to school choice.  Their ideal model has the following central features:

1. Choice programs should not allow private schools to use their regular admissions standards and procedures.  Instead, they should be requires to accept all applicants or use a lottery if over-subscribed.
2. Participating schools should be required to accept the voucher amount as payment in full even though that amount is almost always less than their regular tuition, less than their cost to educate each student, and far less than what is provided to students in traditional public schools.
3. Choice programs should focus on low-income students in low-performing public schools.
4. Participating private schools should be required to administer and report results from the state achievement tests.

This model is not a program that powerful backers of school choice would be willing to accept.  It is their ideal.  It is the starting point for their political negotiations, not what they would be willing to accept after compromises to win political support.

Why do they favor all of these regulations?  I think they have four main arguments:

1. State funds require accountability to the state for performance.
2. Regulation protects kids and improves outcomes from choice.
3. Regulation improves the political prospects for choice.
4. Achievement tests are a reasonable proxy for school quality that a regulator could use to decide which schools should be included or excluded from the set of options available to parents.

I think all four of these arguments are mistaken.  In subsequent posts I’ll consider and rebut each of them.

(Guest Post by Lindsey Burke)

Tulane’s Doug Harris found himself at a crossroads in the Austin American Statesman earlier this week, writing:

The school-reform movement stands at a crossroads. One camp wants unfettered free markets, while charter school leaders and others want to offer families choice and preserve meaningful oversight and accountability.

We’re certainly at a crossroads, but road signs look very different from where I stand. I see one road that leads to parents empowered to choose education options that work for their children, with schools that are held directly accountable to those parents, and another road (perhaps it winds through New Orleans) that puts up regulatory barriers on school choice options and creates no genuine accountability to speak of.

Who Knows Best? Families vs. Bureaucrats

Harris argues in the Statesman that “free markets don’t make sense for schools” because “families expect schools to do a lot of things for their children — teach academic skills, social manners and good values — most of which families don’t have good information about.”

But is it true that families don’t have good information about these facets of schooling? Or, to put it differently, that the government is in a better position to evaluate and make decisions about these difficult-to-quantify outcomes?

Markets produce voluminous information about goods and services that answer the questions consumers are actually asking about a particular product. The oft-referenced ESA Yahoo message board that families in Arizona established after the introduction of the education savings account option in 2011 is a good example.

Current and potential Empowerment Scholarship Account families of Arizona meet here and share ideas and resources for how to best acquire, keep and utilize the funding they need for their child’s individual education.

This is an informal, unaffiliated parent information group where we hope to share ideas, questions and information with each other as we make exiting, individual educational decisions for our special needs, military, D/F school, foster/adoptive children and grandchildren.

Although we list the official ESA website and may share many resources here, we are an informal group of parents and grandparents and are unaffiliated with any formal government or private organization.

As Julie Trivitt and Patrick Wolf have identified in their work on school branding, Catholic schools created a ‘corporate brand’ that signals to parents engaged in the school selection process that their schools provide a religious education and academic quality. This type of branding provides informative shortcuts for parents as they work to choose a school that meets the needs of their child. And critically, when a brand fails to accurately reflect a school’s attributes or quality, inaccurate brands become “an instigator of programmatic attrition.”

Not only do parents have the most intimate perspective on the needs of their own children, but they also tend to be savvy consumers of education services and products, which is why parents leave a provider when brand promises are not met. Choice increases parental involvement, introduces parent-driven decision making, and produces consumer information that is far more detailed (and actionable) than accountability measures in place in a government-run K-12 education system.

By contrast, district schools provide answers to how students perform (the answer usually being, not too well), using blunt measures largely based on state and national tests, and do little if anything to hold those in charge accountable for underperformance. As Matt Ladner has demonstrated, although just three in 10 students in eighth grade in Texas public schools are proficient in reading, 92.5 percent of school districts received a “met standard” designation, with just 6.5 percent of districts receiving a “needs improvement” label. Who’s being held accountable there?

Yet in a market – say, a robust ESA market – consumers not only have more useful information available to make informed choices that meet the needs of their children, they can hold providers to account for not meeting promises. As Jason Bedrick has noted, “real accountability means being directly accountable to those who bear the consequences of your performance.”

Measuring Quality: But By Whose Measure?

Harris also goes on to question the wisdom of choice without the omnipotent hand of the government regulating accountability. “Even if free markets did work well,” he says, “it would be reasonable for policymakers to ask for some measurable results. It’s hard to think of another case where government writes checks to private organizations without checking whether taxpayers are getting anything for their money.”

First, ESA and other education choice funds do not go to “organizations.” Funds go to families, not schools. Schools certainly benefit, by only by way of parents taking their funds to schools that fulfill what they’re looking for. Likewise, food stamps are for the hungry, not grocery stores; Section 8 housing vouchers are for those who need shelter, and are not subsidies designed to prop up the apartment building industry.

Second, the government regularly writes checks to individuals for use at a variety of organizations without requiring either those individuals or organizations to meet certain government-imposed metrics. Grocery stores accepting food stamps aren’t held to higher standards than those than don’t, nor are food stamp recipients required to abide by any dietary guidelines or limited to a certain caloric intake. Contra Harris, this approach is the norm for nearly every entitlement and welfare program, including Social Security, SNAP, WIC, Section 8, and so on. As Jay has noted, the feds aren’t checking on grandma to see that she spent her social security money on vegetables or rent.

This is the norm in education policy as well. Pell grants to colleges require accreditation, but that is far from a measure of academic quality. Colleges that accept Pell grants are not required to administer national tests or any tests at all. Nor are they required to meet government-imposed benchmarks for graduation rates or any other quantifiable measures, let alone to harder-to-quantify ones like civic values or noncognitive skills.

Third, and more germane to the choice conversation, is Harris’s notion that government is needed to ensure accountability. Not only are government regulations in education a far inferior form of accountability than market driven mechanisms, but they can actually have the inverse effect of what was intended by regulation-hawks. And coming from Louisiana himself, where the high-regulation model is in place (requiring private schools accepting students on a voucher to take the state test and punishing “underperformers” by kicking schools that parents have chosen out of the options pool), Harris should acknowledge that the so-called accountability regulations have not lived up to their proponents’ promises and may have had the exact opposite effect of what was intended.

Heavy-handed regulations (a state testing mandate, among others) have discouraged the vast majority of private schools from participating, while likely encouraging lower performers (as indicated by student attrition from those schools prior to entering the voucher program) to join the LSP, willing to incur the regulations in order to secure a new funding stream.

Fairness: Market Mote and Government Beam

Harris then goes on to argue that a free market in education wouldn’t be fair:

In the average free market, wealthier people get higher quality items while low-income families get the lowest. That might be tolerated when we are talking about buying breakfast cereal at the grocery store — but not when we are talking about schools.

To reiterate the argument choice proponents have been making ad nauseam since 1955, this is exactly the system that is currently in place. Wealthier families can currently pay for private options that are higher quality or meet their needs better than their assigned public school, or can purchase a home in a district that reflects their education preferences.

One concern Harris voices that is a valid discussion to have is whether vouchers could create a price floor for tuition prices. This is another reason why ESAs have advantages over traditional modes of school choice. The ability of parents to roll over unused funds year-to-year and to direct dollars to multiple services and products and providers mitigates this issue to a large degree. But that’s hard to see if you refuse to acknowledge that ESAs are functionally different than vouchers.

At the Crossroads: Assessing the Evidence

Harris concludes his op-ed with one last argument about what the research supposedly says:

The research lines up with what basic economics predicts. Across many studies, students using vouchers end up with lower achievement levels than those in traditional public schools. The effects have been especially bad in states like Louisiana and Ohio, where voucher programs are most similar to Senate Bill 3.

Actually, the school choice literature shows an excellent track record, that the government regulatory approach should be envious of. There have been 15 gold-standard, random-assignment evaluations of private school choice programs. Ten of those found statistically significant increases in academic outcomes, three found no difference, and two were negative. Those two negative evaluations were both from Louisiana, and were likely due to the uniquely prescriptive regulatory environment in Harris’s Pelican State.

Harris is indeed right that the school reform movement stands at a crossroads: we can either overregulate choice in a way that limits participation and basically replicates the public system, or we can allow choice and innovation to flourish by trusting families. If we want something different, it’s time to take the road less traveled.

(Guest Post by Jason Bedrick)

At the risk of arousing Greg’s ire, I’d like to talk about the market for education.

In a recent interview, Professor Joshua Goodman of Harvard thoughtfully expressed skepticism that markets could improve quality in education. Whereas I’ve found other attempts to make the “education is different!” argument to be unpersuasive, Goodman makes a few solid points that choice advocates would do well to consider:

I became an economist in part because of Milton Friedman’s argument that vouchers could improve schooling through market forces. At the time, this argument struck me as both revolutionary and obviously right. Competition improves supermarkets, restaurants — why shouldn’t this model apply to schools? It seemed to me that anyone who denied this idea didn’t understand basic economics.

But the more I read, the more I realized that the empirical evidence for choice and market forces improving educational outcomes is thin at best. I found that disappointing and also puzzling, and I have spent some time thinking about why that theory doesn’t match current reality.

Regarding the evidence, it’s certainly true that none of the catastrophes that school choice critics have predicted have come true. But neither have the policies produced the transformative changes that supporters had promised. Instead, we’ve had a consistent batch of random-assignment studies finding positive but relatively small effects (setting aside the high-regulation disaster in Louisiana). The effects tend to be larger for minorities, parents are much happier, and they cost less per pupil, so I think they’re still worth pursuing even on purely consequentialist grounds, but school choice programs have not been transformative. Why not? Goodman continues:

Here’s what I think the biggest problem in thinking of schools as a classical market. Econ 101 models assume consumers observe product quality. But schools are complicated goods, and quality, particularly a school’s long-run quality, is hard to judge for many parents. It takes a lot of time to figure out whether this school and these teachers are serving my child well. Unlike restaurants or supermarkets, where quality can be judged at the moment of the purchase, school quality reveals itself later. […]

Observing school quality is quite costly, and in many settings there is no credible way to inform future students about the quality of education they are getting. The for-profit college sector is a perfect example of this. Market forces fail to discipline for-profit colleges because for an individual student there’s no repeated game here. Students enroll and only much later realize lousy labor market outcomes. In particular, that students must enroll for a while to see long-run outcomes limits the power of the market to provide discipline. The time it takes to learn that a school is low quality damages the student, and that student’s information may not be transmissible to other students in a systematic or credible way.

Goodman concludes that he does believe “schools could use a bit of market pressure to counteract inertia” because district “monopolies clearly allow many schools to coast on their current trajectories without trying new approaches or investing more effort,” but he is “much more skeptical now than I was before that market forces are some sort of panacea, as they appear to be in some industries.” In the end, he thinks there will “always be a need for public accountability, if only to lower information costs to parents and students.”

I agree with much of this analysis. As I wrote at EdNext today:

There is no panacea. There is no perfect information just as there are no perfect bureaucrats or, for that matter, perfect parents. The question before us is how to design a system with imperfect people and imperfect information that will come as close as possible to providing every child with access to a high-quality education.

Getting parents the information they need to make good decisions is indeed a challenge. As Goodman notes, it is harder to identify a quality education than it is to identify a good meal or a good car. There is a great degree of subjectivity and the payoff is usually far in the future.

The question then becomes what sort of institutions are better equipped to address this challenge. As Lindsey Burke and I argue in a new report published by the Heritage Foundation and the Texas Public Policy Foundation, the technocratic approach of holding schools accountable on metrics that are easily observable and standardized creates perverse incentives that narrow the curriculum, stifle innovation, and can drive away quality schools from participating in the choice program. Instead, what we need are a variety of different third-party reviewers and platforms for parents and students to share their personal experiences in order to provide parents with the information they need.

I won’t rehash the entire argument here. You can either read the report on the short version at EdNext. The main point I wanted to highlight here was that the market is more likely than the government to overcome the information challenge — but for that to happen, there needs to be a large enough market. As I noted:

The K-12 education sector has historically lacked high-quality sources of information about school performance, but to a large extent that is because the vast majority of students attend their assigned district school. With little to no other educational options, there has been little parental need for information to compare competing options. And without much in the way of competition, existing private schools don’t feel great pressure to be forthcoming about performance data. However, as states implement educational choice policies, the demand for information will increase and schools that refuse to share their data will be at a competitive disadvantage. We are already seeing parents to turn organizations like GreatSchools.org and Niche.com to find information about schools they are considering and we should expect to see more organizations emerge as demand increases.

School choice programs that merely fill empty seats are not enough. They won’t bring the transformative change that Milton Friedman and others predicted. Modest choice programs produce modest results. It’s time to go bold.

Doug Harris has a new post that attempts to reply to the many critics of his New York Times op-ed, including me.  In the NYT piece Harris claimed that “one well-regarded study found that Detroit’s charter schools performed at about the same dismal level as its traditional public schools.” He also claimed that the relatively light regulatory approach to Detroit’s charter schools has led to “the biggest school reform disaster in the country.”  He prefers instead the heavy-regulation approach adopted in New Orleans, which he says has produced “impressive” results.  He then suggests that failing to believe these claims is “a triumph of ideology over evidence.”

Several critics, including me, noted that the “well-regarded study” Harris cites actually finds that Detroit charter schools are producing significantly greater gains than traditional public school alternatives — gains that are only slightly smaller than those in New Orleans and greater than in another high-regulation darling, Denver.  But Harris wants to continue posturing as the person backed by science and evidence, while describing his opponents as ideologues.  In his reply to critics in Education Next he uses the word “evidence” 16 times.

So, his defense of the claim that Detroit charters perform “at about the same dismal level as its traditional public schools” must be based on evidence, right? Actually, no.  He provides four arguments to rescue his assertion that “the failure of Detroit charter schools to improve student outcomes” is true despite the fact that the CREDO evidence he cited to make that claim shows otherwise.  Each one of those four arguments is based on assumptions, not evidence.  If you believe all of his assumptions, you might believe his claim that Detroit charter schools really are a disaster. But drawing conclusions that depend completely on assumptions and are contrary to the evidence he cites is certainly not science.  It seems more like a faith-based or ideological exercise.

Let’s consider his four arguments.  First he says, “given the lack of oversight in Detroit and evidence from other cities that some charter schools cherry-pick their preferred students, these results may make Detroit’s charter schools look better than they are.”  Got that?  He has no evidence that Detroit charters are cherry-picking students at all, let alone that they are doing so at a higher rate than in New Orleans, but he nevertheless posits that “if it’s happening, then the charter effects on achievement [in Detroit] are inflated.”

Second, given Harris’ assumed concerns about cherry-picking in Detroit charter schools and the inability of the CREDO study to account for that, he examines evidence from the urban NAEP test and finds that the city of Detroit has experienced below average growth in those scores in recent years.  Using NAEP results from the entire city to draw conclusions about Detroit’s charter schools requires a host of assumptions.  He’d have to assume that test results from all schools, charter and traditional, somehow speak to the effectiveness of charter schools.  He’d have to assume that demographic and other non-school factors in Detroit do not affect the comparison of test growth in Detroit relative to other cities.

A number of education analysts have coined a term — misNAEPery —  to capture how unreasonable it is to make the assumptions required to use NAEP to compare policies across jurisdictions.  Oddly, some of those analysts who like to accuse others of misNAEPery, like Morgan Polikoff and Matt Barnum, have somehow failed to denounce Harris’ use of misNAEPery in both the NYT op-ed and in the Ed Next reply to critics.  Both have even “retweeted” Harris’ new post, so we can assume they’ve read it.

And just to anticipate concerns about my own consistency on the use of NAEP, I think comparisons using NAEP that control for observed demographics are about as convincing as the CREDO results, which also rely on comparisons controlling for some observed characteristics.  Information from NAEP or CREDO can be interesting or suggestive, which is why I say that Arizona charters “appear” to be doing very well, even if I am not convinced that any of this is causal.  At the very least, it is useful to offer a disclaimer that neither NAEP nor CREDO provides convincing causal evidence, even if confession does not assure absolution.  Harris does not offer any disclaimer and instead uses his NAEP comparisons to bolster his assumption that Detroit charters may be cherry-picking.

Third, Harris builds on the observation that Detroit city has very low NAEP test scores to assume that “the extraordinarily low standing of the city as a whole, to the degree it is caused by low performance of traditional public schools, should make it easier to improve student outcomes when trying something new.”  It is also quite plausible — perhaps more plausible — that a city with extraordinarily low test scores also has severe social and economic problems that are outside of the control of schools, which would make it harder for charters to improve outcomes.

Lastly, Harris argues that even if Detroit charters have produced gains, the gains are smaller than those in New Orleans, so we should prefer the regulatory approach used in New Orleans to the one in Detroit. But this assumes that any greater gains produced by New Orleans charter schools are caused by the regulatory approach in that city.  In fact, we have no idea whether New Orleans’ regulations helped, hurt, or had no effect on how large the gains in that city were. For all we know, the gains made by New Orleans charters are largely attributable to the importation of top-notch human capital from elite colleges and a huge increase in per pupil spending, and that these gains were made despite the hindrance of burdensome regulations.  The heavy regulations in Denver somehow failed to produce gains as large as those in Detroit.

Other than these four-assumption-dependent arguments, Harris offers very little to defend the strong claims he made in the NYT that Detroit charter schools have been a “failure” and a “disaster.”  He does cite some national “charter-friendly” organizations, like CRPE and NACSA,  as being critical of Detroit charters.  But that is an argument from authority — not evidence.

He also falsely suggests that Detroit has failed to close failing charter schools.  In fact, 30% of Detroit’s charter schools have been shuttered.  And another national charter-friendly organization, NAPCS, gives Michigan higher marks on closure than Louisiana, noting that Michigan has closed more charter schools than Louisiana, 47 to 26.

Harris also cites two voucher studies with negative results — one of which is an RCT and the other not — as proof that lower regulation approaches are less effective.  Leaving aside the distinct possibility that the negative RCT result for Louisiana was actually a function of the over-regulation of that program, it’s important to note that Harris fails to cite the entire literature of rigorous studies on the effects of private school choice programs, which overwhelmingly shows positive outcomes.

I agree with Doug that ideology, which could more kindly be described as “principles” or “values,” has an important role to play in policymaking.  I just disagree with him that the evidence clearly shows the superiority of a high-regulation approach to school choice.  I think it’s more accurate to say that the evidence is unclear on this matter, which means that we may — appropriately — need to rely more on our values, principles, and broader ideology when deciding how to proceed.

[Edited to remove tangential comment that was poorly phrased.]

The Gates Foundation’s education reform strategy is in the midst of the most catastrophic failure since the Annenberg Foundation blew $500 million in the 1990s.  The wheels are coming off Common Core, the center-piece of the Gates reform strategy.  Today’s front-page Wall Street Journal article documents how states and districts are abandoning the standards and their aligned tests and/or backing away from making the necessary expenditures to implement the new standards.  At this point only 23 states are still using one of the two Common Core assessments, putting me clearly in the lead on the Greene-Polikoff Wager.  The WSJ article paints a devastating picture of Common Core’s collapse.

Even local efforts by the Gates Foundation to implement its teacher quality strategy are falling apart.  Gates pledged $100 million to the Hillsborough School District in Tampa, Florida to make it the model of its reform strategy.  As the district is running out of Gates money and discovering the unsustainability of its own financial commitments, the whole effort of using new teacher evaluation methods, mentoring, and merit pay is about to be dismantled.

Despite all of this investment, Hillsborough is getting lousy academic results.  As can be seen in the table below, Hillsborough has been doing very poorly on the US Department of Ed’s Urban NAEP over the last 2 and 4 years.  Hillsborough has even lagged far behind the generally disappointing national results.  Nor is this a Florida problem as Hillsborough lags far behind the trend in Miami.  Of course, one cannot attribute these aggregate trends to any specific policy, but political interpretations do not hinge on these methodological niceties.  The obvious conclusion policymakers are drawing is that the Gates effort in Hillsborough cost a fortune, is not financially or politically sustainable, and is an academic flop.

The question is whether philanthropists and ed reformers are going to learn the right lessons from the unfolding Great Gates Political FAIL.  Some seem to have mistakenly concluded that the problem was just poor communication and messaging.  Others seem to think we just need to try harder to succeed with implementation.  I’m convinced that a top-down strategy that falsely invokes science to identify “best practices” and then attempts to impose those practices on our highly decentralized education system is always doomed to fail, regardless of how it is “messaged” and no matter how earnest we are about implementation.

There are no indications as of yet that folks at the Gates Foundation or other major reform organizations have learned the proper lessons.  Vicki Phillips just announced that she is leaving as the head of Gates education efforts, but the Foundation’s public statements indicate no change in strategy.  It’s unclear whether Phillips is leaving because of perceived failure, because of the repeated mis-representation of research, or just because it is time for a change after 8 years at the helm.  And other foundations seem to be drifting toward more top-down, high-regulation approaches.

Effective philanthropy is hard and education reform is even more challenging.  But unless the major organizations change their approach they are doomed to repeat this failure.

I’ve been having a debate over the last few weeks with Neerav Kingland and others about the dangers of a high-regulation approach to school choice.  (You can see my posts on this so far here, here, here, here, here, and here).  I know this seems like a lot of posts on a topic, but as one grad student observed, it took more than 100 pieces about the obvious error of government reported high school graduation rates before people fully acknowledged the error and too significant steps to correct it.  Let’s hope convincing ed reform foundations and advocates to scale way back on their infatuation with heavy regulation does not require the same effort as moving more obstinate and dim-witted government officials.

One heavy-handed regulatory approach that is particularly worrisome is the strategy of portfolio management.  Under this strategy, a portfolio manager, harbor master, or some other regulator actively manages the set of school options that are available to families by closing those believed to be sub-par and expanding or replicating those that are thought to be more effective.  This approach is being implemented in New Orleans and the city appears to be experiencing significant gains in achievement tests, so Neerav and others are puzzled as to why I don’t support it.

I’ve tried to express my reasons for opposing portfolio management in several ways.  I tried mocking it: “If education reform could be accomplished simply by identifying and closing bad schools while expanding good ones, everything could be fixed already without any need for school choice.  We would just issue regulations to forbid bad schools and to mandate good ones.  See?  Problem solved.”  That clearly didn’t work because folks like NOLA advocate Josh McCarty replied: “moving the left end of the performance curve to the right through regs has gotten more kids in higher perf schools.”

So, let me try again.  A portfolio manager can only move “the left end of the performance curve” if the regulator can reliably identify which schools are likely to harm students’ long-term outcomes and which ones are likely to improve them.  If you don’t really know whether schools are on the left or right end of some curve of quality, closing schools just limits options without improving long-term outcomes.  But backers of portfolio management are not lacking in confidence.  They have achievement test results, so they think they know which are the good and bad schools.

Unfortunately, they are suffering from over-confidence.  Achievement tests are useful but they are not nearly strong enough predictors of later life outcomes to empower a portfolio manager to close a significant number of schools because he or she “knows” that those schools are “bad.”  In fact, the research I reviewed on rigorous evaluations of long-term outcomes from choice programs suggests that using test scores to decide whether a bunch of schools should be closed or expanded would lead to significant Type 1 and Type 2 errors.  That is, in their effort to close bad schools, portfolio managers may very well close schools with lower test performance that actually improve high school graduation, college-attendance, and lifetime earnings.  And they may expand or replicate schools that have high test performance but do little to improve these later life outcomes.

If there were an active portfolio manager of Florida charter schools, they would have closed a bunch of charter schools that were doing a great job of improving students’ later life outcomes.  As Booker, et al’s research shows, relying solely on test scores to distinguish good from bad schools would lead to serious errors by an active portfolio manager:

The substantial positive impacts of charter high schools on attainment and earnings are especially striking, given that charter schools in the same jurisdictions have not been shown to have large positive impacts on students’ test scores (Sass, 2006; Zimmer et al., 2012)…. Positive impacts on long-term attainment outcomes and earnings are, of course, more consequential than outcomes on test scores in school. It is possible that charter schools’ full long-term impacts on their students have been underestimated by studies that examine only test scores. More broadly, the findings suggest that the research examining the efficacy of educational programs should examine a broader array of outcomes than just student achievement. (pp. 27-8)

Conversely, foundations and portfolio managers are pouring more resources into certain types of schools with strong test performance that are failing to show much benefit for students’ long-term outcomes.  As Angirst, et al, Dobbie and Fryer, and Tuttle, et al show, a bunch of charter schools with large achievement test gains, including Boston “no-excuses” schools, Harlem Promise Academy, and KIPP, have produced little or nothing in terms of high school graduation and college-attendance rates.

Portfolio management guided solely by test scores would seriously harm students by unwittingly closing a bunch of successful schools, like those Booker, et al studied in Florida, while expanding and pouring more resources into ones with less impressive long-term results, like those studied by Angirst, et al, Dobbie and Fryer, and Tuttle, et al.

Matt Barnum challenged me on Twitter to describe what evidence would persuade me to support portfolio management.  At a minimum I would want to see that portfolio managers have reliable tools for predicting long-term outcomes for students so they knew which choice schools should be closed and which should be expanded or replicated.  The evidence I’ve reviewed here and in more detail in this prior post suggests that they do not have a reliable tool and so the entire theory of portfolio management falls apart. I’m not making the strawman argument that test scores are useless or that no school should ever be closed by regulators.  I’m just arguing that portfolio management requires confidence in the predictive power of achievement tests that is not even close to being warranted by the evidence.

But what about the impressive achievement gains that Doug Harris and his colleagues find are being produced in New Orleans?  Let’s keep in mind that many reforms have been implemented in New Orleans at the same time.  Even if we were confident that the test score gains in New Orleans are not being driven by changes in the student population following Katrina (and Doug and his colleagues are doing their best with constrained data and research design to show that), and even if these test score gains translate into higher high school graduation and college attendance rates (which Doug and his colleagues have not yet been able to examine), we still would have no idea whether portfolio management and other high regulations in NOLA helped, hurt, or made no difference in producing these results.  In fact, the evidence from the 7 rigorous studies on school choice programs with long-term outcomes suggests that portfolio management and other heavy regulations are neither necessary nor desirable for producing long-term gains for students.

Neerav, Matt Barnum, and Josh McCarty have suggested that I am making overly-broad claims not consistent with evidence.  I think the opposite is true.  I’ve carefully cited and quoted the relevant research and drawn the obvious conclusion — active portfolio management based on achievement tests is likely to make harmful errors and unnecessarily restrict options.  In fact, it seems to me that the burden is on supporters of portfolio management to demonstrate that they are able to reliably distinguish between schools with good and bad long-term outcomes.  If you are going to go around telling families that they can’t choose a certain school because it is bad for them, you had darn better be confident that it really is bad.

Neerav Kingsland is smart, honest, and, in a positive development for ed philanthropy, is now leading education efforts at the Arnold Foundation.  I enjoyed his response to my series on the dangers of high regulation of school choice (it started here, and then had parts 1, 2, 3, and 4).  But the bulk of the critical response from Neerav and others seemed to focus on defeating a strawman rather than what I actually wrote.  They depict me as arguing against any regulation when my post was explicitly against “high-regulation” of school choice.

Maybe the mistake was on my part.  Maybe I just wasn’t clear enough, so let me succinctly reiterate here my concerns.  Some of the big reform-oriented foundations have drifted toward a high-regulation approach to school choice that I think is very dangerous and counter-productive.  Many of the features of this high-regulation approach can be seen in Louisiana and New Orleans.  They are:

1. Families should have choices, but they should only choose among quality schools.  So, a portfolio manager, harbor master, or other type of regulator should use test scores to identify who is and is not a quality school operator and eliminate from the set of options a large number of schools that appear to be sub-par.
2. All choice schools should administer state achievement tests so that regulators and families can more easily make comparisons.  Besides, it is normal to expect that government funds bring with them the requirement to demonstrate performance to the government.
3. To ensure equity of access, choice schools should not be allowed to use their own admissions criteria but should be required to take all applicants or admit by lottery.  They should also be required to accept the amount of state funding as payment in full.  And equity would be further enhanced if we targeted choice programs toward low income students in low performing traditional public schools.
4. If private schools are reluctant to go along with this high-regulation approach, maybe it is best just to concentrate on charter schools which have no alternative but to accept whatever regulations come with state funding.  Besides, private schools seem to bring with them other political problems and things uncomfortable to foundations, like religion.

In describing this high-regulation approach increasingly preferred by the big reform foundations I don’t think I am making a strawman out of their views.  This seems like a fair summary of what they believe and it describes what they have put into practice in choice programs in New Orleans and Louisiana.

But I think my objections are being turned into a strawman.  Neerav asks in the title of his post: “To Regulate or Not?”  Joel Klein similarly tweeted: “read you as saying choice per se, no regulation, is solution.”  This is not what I argued.  My argument was against high-regulation, not all regulation. Here is what I was actually trying to argue:

1. Test scores are useful but are not strong enough predictors of later life outcomes to determine which are the “quality” schools that should be among the options available to families.  So, regulators relying on test scores will experience false positives and false negatives if they try to actively manage a portfolio of schools.  This doesn’t mean that regulators should never close a school or should ignore test results (both of which are strawman arguments), but they should be significantly more humble about what they really know about school quality.  It means that they should give a fair amount of deference to parent preference and should only eliminate schools from choice programs if there are multiple indicators of failure, and exercise some human judgment about those indicators.  In the end I think I agree with Ashley Jochim who argued: “It may be that government will be more effective at establishing performance floors – much like they do in the arena of auto safety – rather than driving continuous improvement in school quality.”
2. High regulation does not improve equity of access because it drives away many of the highest quality schools.  Heavy regulation caused two-thirds of the private schools in Louisiana to refuse to participate in the state’s voucher program, including most of the best private schools in the state.  Neerav is right that I should have addressed how equity could be achieved without these regulations. Rather than trying to compel equity of access through regulations that instead drive schools out of the program, we should incentivize equity by having student-weighted voucher amounts.  That is, students who are more expensive to educate (disabled, ELL, low-income) should be provided with larger voucher amounts so that schools are compensated for taking on more expensive to educate students.  Most of our concerns about equity of access are likely to be addressed if we simply empower disadvantaged families with higher resource amounts.
3. Focusing only on charter schools because they are more compliant with a high-regulation approach is a serious mistake.  The evidence suggests that private school choice programs may have stronger later-life outcomes for students than charters.  But more importantly, I argued: “No one knows the ideal political strategy or regulatory scheme, so having a variety of different approaches [vouchers, charters, ESAs, etc…] allows us to learn about how these different methods for expanding choice are doing.  We need choice among choice.”

Just as we should be humble about using test scores to identify quality schools, we should be humble about knowing the ideal political or regulatory strategy.  While I don’t exactly know the right regulation for choice, I’m pretty confident that the high regulation strategy being pursued in LA and New Orleans is a really bad idea.  It deters quality schools from participating, it unnecessarily excludes schools from being options, and it is counter-productive in how it interferes with the operations of choice schools.

In my final piece of this series arguing against the high-regulation approach to school choice, I’m going to discuss whether achievement test results are a reasonable proxy for school quality.  Achievement tests are at the center of the high-regulation approach.  They are used by regulators — whether authorizers, portfolio managers, or harbor masters — to identify good and bad schools, to determine whether they should be included as choice options, and to shape the goals schools should pursue.

There is no question that growth in student learning provides us with some useful information.  The problem is that school quality is much broader than just test score results.  I always understood that achievement tests were only a partial and imperfect indicator of school quality, but I used to believe that other aspects of school quality not captured by achievement tests were largely correlated with those test results.  That is, I used to think that if a school raised scores it probably meant that students were safer, more students would graduate, more students would learn productive values, and more students would go on to become successful adults.

Unfortunately, the evidence is increasingly clear that test scores are only weakly correlated with all of these other desirable outcomes from schools.  All you have to do is look at yesterday’s post.  Schools that produce the largest achievement test gains are not necessarily the ones that produce higher graduation, or college-attendance rates.  And sometimes schools with unimpressive achievement gains make significant contributions to attainment and annual earnings when students join the workforce.  I used to think that this couldn’t be possible.  All of these happy outcomes had to be aligned.  They just aren’t.

If you are not persuaded by the evidence I reviewed yesterday on the disconnect between achievement results and other outcomes, I suggest you read an excellent book written by Nobel Prize winning economist James Heckman and his students called The Myth of Achievement Tests.

The problem is that the high-regulation approach needs achievement tests to be correlated with all of these other good outcomes.  If they are going to pick the school choice winners and losers based on test scores, then test scores need to be strongly predictive of other things we care about.  People have been very slow to accept the fact that test scores are only weakly correlated with later life outcomes because it would be so convenient if readily available and relatively inexpensive test scores could capture something as complex as school quality.  The fact that they don’t throws a monkey wrench into the entire high-regulation machinery.

The reality is that the average low-income parent has more complete information about their kid’s school quality than does the highly-trained regulator armed only with test scores.  When we wonder why parents are choosing schools that regulators and other distant experts deem to be “bad,” it is almost certainly because the parents know more about what is good and bad than do the experts.

The wrong response to recognizing that test scores fail to capture school quality sufficiently is to increase the set of high-stakes measures we collect.  We can’t fix the limits of math and reading achievement tests by adding mandatory “grit” surveys or other measures.  Even informed by a variety of measures, Chinese officials are no more effective in telling state-controlled banks how to allocate capital than portfolio managers are in determining how to allocate school options.  Decentralized decision-making is simply better than central planning.

The school choice movement has to remember that choice is what makes this reform work, not the regulation.  I’m perfectly willing to accept that some regulation is necessary and inevitable.  And I’m willing to make compromises to get programs adopted.  But the cardinal sin of the high-regulation school choice folks is that they believe that heavy regulation is the ideal and should be the starting point for political compromises.

In this series of post against the high-regulation approach to school choice, I have demonstrated that performance accountability is not typical of government programs and that heavy regulation drives away quality supply, hurting rather than protecting the students these regulations are meant to help.  If high-regulation is not the norm and does not help children, supporters of this approach might still favor it if they think it has certain political advantages.

For those interested in private school choice, two political advantages are claimed: 1) High-regulation addresses some  objections, winning votes among skeptics to improve the political prospects of passing and sustaining those programs; 2) High-regulation protects private school choice programs from the political damage caused by scandals and embarrassing outcomes.

Neither of these arguments is supported by experience.  Conceding regulatory measures to skeptics and opponents has hardly changed a single vote.  Backers of the Milwaukee voucher program thought they would get relief from legislative opposition if they accepted more burdensome regulation.  No votes have changed as a result and the program remains as precarious as ever.  Nor has regulation protected programs from scandal.  Judging from the steady stream of news reports about teachers in traditional public schools sleeping with students, it appears that no amount of background checks or government oversight can eliminate rare but regular instances of misconduct.  I’m not arguing against a reasonable and light regulatory framework, I’m just suggesting that higher levels of regulation provide little or no additional political protection.  Determined opponents can always find scandals to exploit and cannot be appeased with anything short of preserving the traditional public system.

I’m actually more worried that key backers of school choice are starting to abandon private school choice and focus all of their energies on charters.  High-regulation is the norm in charter programs.  You don’t have to worry about charter schools refusing to participate in a heavily regulated program since they have no alternatives.  And charters seem to be flourishing.  Charter programs exist in more states with more schools serving more students than do private choice programs.  Many important backers of school choice seem to believe that charters are also getting better results.  As Neerav Kingsland of the Arnold Foundation tweeted yesterday: “why is it the over-regulated charter sector that has had the most breakthroughs with low income students?”

Unfortunately, Neerav is mistaken.  Charters are not producing better results than private school choice.  High-regulation comes with a cost to quality.  Let’s consider rigorous evidence on how charter and private school choice affect educational attainment.  For reasons I will discuss at greater length in the next post, I think attainment is a more meaningful indicator of long-term benefits than achievement test results.  I’m aware of 4 rigorous studies of the effect of charter schools on attainment.  The general pattern among them is that programs producing large gains in achievement test outcomes are producing little or no increase in educational attainment.

Angrist, et al examined Boston charter schools and found significant benefits for charter students on MCAS, SAT, and AP performance.  On attainment they write:

Does charter attendance also increase high school graduation rates? Perhaps surprisingly given the gains in test score graduation requirements reported in Table 4, the estimates in Table 7 suggest not. In fact, charter attendance reduces the likelihood a student graduates on time by 12.5 percentage points, a statistically significant effect. This negative estimate falls to zero when the outcome is graduation within five years of 9th-grade entry. (p. 15)

Nor are results much better for attending college: “While the estimated effect of charter attendance on college attendance is positive, it is not large enough to generate a statistically significant finding.” (p. 16)  Angrist, et al do find a significant shift of students from attending 2 year to 4 year colleges, but we don’t know yet if that shift represents a positive development until we see whether they complete their degrees.  Shifting students to 4 year college for which they are ill-suited and from which they drop out does them no favor.

Dobbie and Fryer examine the results of a single charter school in Harlem, the Promise Academy.  Like Angrist, et al, they find large achievement test gains but little benefit for attainment.  Dobbie and Fryer find a higher high school graduation rate after 4 years of the start of 9th grade, but it disappears by 6 years. (p, 18)  College attendance benefits are also fleeting: “Similar to the results for high school graduation,however, control students eventually catch up and make the treatment effects on college enrollment insignificant.”  Dobbie and Fryer similarly find a shift toward 4 year colleges, but again this result is ambiguous. Four year college should help students obtain more schooling but they report “The number of total semesters enrolled in college between lottery winners and lottery losers is small and statistically insignificant.” (p. 19)

Tuttle, et al’s recent evaluation of KIPP charter schools also finds large achievement test gains for charter students but little or no attainment benefit.  Tuttle and her team at Mathematica make two types of comparisons to assess the progress of KIPP high school students.  In one they find: “For new entrants to KIPP high schools, we also examine the probability of graduating within four years of entry. We find that this group of KIPP high schools did not significantly affect four-year graduation rates among new entrants.” (p. 36)  When they examine students who continued from KIPP middle schools into KIPP high schools, they find a small but statistically significant drop in the rate at which students drop out — about 2 percentage points. (p. 39)

Booker, et al examine charter schools in Chicago and Florida and find significant benefits in educational attainment as well as higher earnings later in the workforce — at least for Florida charter students.  They write: “In Florida, the charter high school students show a consistent advantage in absolute terms of 8 to 11 percentage points from high school graduation through a second year of college enrollment.” (p. 22)  On later earnings they find: “Charter high school attendance is
associated with an increase in maximum annual earnings for students between ages 23 and 25 of $2,347—or about 12.7 percent higher earnings than for comparable students who attended a charter middle school but matriculated to a traditional high school.”

Before the high-regulation folks get too excited about the Booker, et al results as vindication of their approach, they should note that these charter schools did not produce impressive achievement test results.  Booker, et al write:

The substantial positive impacts of charter high schools on attainment and earnings are especially striking, given that charter schools in the same jurisdictions have not been shown to have large positive impacts on students’ test scores (Sass, 2006; Zimmer et al., 2012)…. Positive impacts on long-term attainment outcomes and earnings are, of course, more consequential than outcomes on test scores in school. It is possible that charter schools’ full long-term impacts on their students have been underestimated by studies that examine only test scores. More broadly, the findings suggest that the research examining the efficacy of educational programs should examine a broader array of outcomes than just student achievement. (pp. 27-8)

In the high-regulation approach, these charter schools might well be identified as the “bad” schools for failing to improve test scores, and yet they are the ones that produce long-term success for their students.  In the high-regulation approach a portfolio manager or harbor master might kick these schools out of the program or restrict their growth for failing to produce achievement gains.

Let’s briefly review the results from the three rigorous examinations of the effect of private school choice on educational attainment.  Unlike the charter research, they all show significant benefits for attainment.  Wolf, et al examined the federally funded DC voucher program.  They found little benefit for voucher students on achievement tests but those students enjoyed a 21 percentage point increase in the rate at which they graduated high school.  Cowen, et al examined the public funded voucher program in Milwaukee and found a 5 to 7 percentage point increase in the rate at which voucher students attended college.  And Peterson and Chingos examined a privately funded voucher program in New York City and found that African-American voucher recipients experienced a 9 percentage point increase in attending college.  There was no significant benefit for Hispanic students.

If the high-regulation folks wanted to ditch private school choice to go all-in on charters, they would be making a horrible mistake.  The evidence suggests private school choice is producing stronger long-term results.  In addition, among charter schools, the kinds of schools that high-regulation folks like the most are the ones producing weaker long-term outcomes.  Focusing only on charters making the biggest achievement score gains would miss those charters with more modest achievement results but truly impressive attainment outcomes.  Charter schools offer the illusion of getting the benefits from choice without too much of the messiness markets.  As it turns out, central planning among charter schools is no easier than central planning among traditional public schools.

In addition to losing quality if key choice backers were to support charters to the exclusion of private school choice, there are obvious political advantages to backing both types of choice.  Private school choice has helped make the world safe for charters by taking more of the political heat.  We wouldn’t have the same expanding charter sector were it not for the credible threat of even more private school choice.  And the choice movement would be wise to spread its bets across a variety of approaches to expanding school choice.  No one knows the ideal political strategy or regulatory scheme, so having a variety of different approaches allows us to learn about how these different methods for expanding choice are doing.  We need choice among choice.

In my last piece in this series against the high-regulation approach to school choice, I observed that accountability to the government does not automatically follow from receiving government funds.  In fact, most government programs, including Food Stamps, Social Security, Pell Grants, and the Day Care Tuition Tax Credit, have no requirements for performance accountability to the state.

Even if government accountability is not the norm for government programs, some people may still favor requiring choice schools to take the state test and comply with other components of the high-regulation approach to school choice, such as mandating that schools accept voucher amounts as payment in full, prohibiting schools from applying their own admissions requirements, and focusing programs on low-income students in low-performing schools.  Some people, including many of the most powerful backers of school choice, seem to believe that these regulations help protect kids and improve outcomes.

Let’s leave aside for now discussion of whether this set of heavy regulation negatively affects the quality of participating schools.  And let’s also leave aside whether these regulations are even effective in promoting equity of access to participating schools for disadvantaged students.  The real problem is that heavy regulation dramatically reduces the number of participating schools.  Arizona’s choice programs have light regulation and near-universal participation among private schools.  Florida’s tax credit program has more regulation, although it does not require taking the state test.  It has almost two-thirds of private schools willing to take students.  But in Indiana’s heavy-regulation program the private school participation rate drops to around 50%.  At least in Indiana, many private schools were accustomed to administering the state test as a requirement for participating in inter-scholastic athletics.  In Louisiana, where the heavy regulation and state-testing requirement were new, only about 1/3 of private schools are willing to participate in the voucher program.  Survey research by Brian Kisida, Pat Wolf, and Evan Rhinesmith confirms that heavy regulation is driving private schools away from these programs.

The only equity of access that is promoted by the heavy-regulation approach is that everyone is equally unable to access schools that refuse to participate in the programs.  In their desire to protect disadvantaged students, the backers of this heavy-regulation approach have ironically done serious harm to these students by driving away most of the supply.  And the minority of private schools that are willing to participate are likely to include many of the lower quality schools.  Who is most likely to be willing to abandon control over their admissions, accept tiny voucher amounts as payment in full for serving the lowest achieving students, and is willing to take the state achievement tests?  Financially desperate private schools with a lot of empty seats are likely to be first in line to accept these terms.  High-quality private schools may at most make a token number of seats available.  Rather than protecting access and ensuring quality, heavy regulation is having the opposite effect.  Heavy regulations are eliminating the bulk of options and especially driving away the highest-quality private schools.

It should come as no surprise to anyone if we see some very disappointing academic outcomes in Louisiana’s voucher program.  A heavy regulation program that some major backers of school choice believe represents the “ideal” approach is actually designed to give us the worst outcomes.  If we do see bad results, the first impulse of the backers of heavy regulation will be to double-down on regulation.  They’ll wonder who the bad schools are and call for regulators to remove them from the program.

If education reform could be accomplished simply by identifying and closing bad schools while expanding good ones, everything could be fixed already without any need for school choice.  We would just issue regulations to forbid bad schools and to mandate good ones.  See?  Problem solved.  But real education reform requires using the power of choice and competition to provide incentives to create more good and to reduce bad.  The whole problem with the high-regulation approach is that it falsely believes regulators can define, identify, and require good outcomes.  If that were in fact possible, we would have already solved the problem and we could have done so without any school choice.  The enduring troubles of the traditional public system tell me that is not possible.