Jay P. Greene's Blog

(Guest Post by Jason Bedrick)

At the risk of arousing Greg’s ire, I’d like to talk about the market for education.

In a recent interview, Professor Joshua Goodman of Harvard thoughtfully expressed skepticism that markets could improve quality in education. Whereas I’ve found other attempts to make the “education is different!” argument to be unpersuasive, Goodman makes a few solid points that choice advocates would do well to consider:

I became an economist in part because of Milton Friedman’s argument that vouchers could improve schooling through market forces. At the time, this argument struck me as both revolutionary and obviously right. Competition improves supermarkets, restaurants — why shouldn’t this model apply to schools? It seemed to me that anyone who denied this idea didn’t understand basic economics.

But the more I read, the more I realized that the empirical evidence for choice and market forces improving educational outcomes is thin at best. I found that disappointing and also puzzling, and I have spent some time thinking about why that theory doesn’t match current reality.

Regarding the evidence, it’s certainly true that none of the catastrophes that school choice critics have predicted have come true. But neither have the policies produced the transformative changes that supporters had promised. Instead, we’ve had a consistent batch of random-assignment studies finding positive but relatively small effects (setting aside the high-regulation disaster in Louisiana). The effects tend to be larger for minorities, parents are much happier, and they cost less per pupil, so I think they’re still worth pursuing even on purely consequentialist grounds, but school choice programs have not been transformative. Why not? Goodman continues:

Here’s what I think the biggest problem in thinking of schools as a classical market. Econ 101 models assume consumers observe product quality. But schools are complicated goods, and quality, particularly a school’s long-run quality, is hard to judge for many parents. It takes a lot of time to figure out whether this school and these teachers are serving my child well. Unlike restaurants or supermarkets, where quality can be judged at the moment of the purchase, school quality reveals itself later. […]

Observing school quality is quite costly, and in many settings there is no credible way to inform future students about the quality of education they are getting. The for-profit college sector is a perfect example of this. Market forces fail to discipline for-profit colleges because for an individual student there’s no repeated game here. Students enroll and only much later realize lousy labor market outcomes. In particular, that students must enroll for a while to see long-run outcomes limits the power of the market to provide discipline. The time it takes to learn that a school is low quality damages the student, and that student’s information may not be transmissible to other students in a systematic or credible way.

Goodman concludes that he does believe “schools could use a bit of market pressure to counteract inertia” because district “monopolies clearly allow many schools to coast on their current trajectories without trying new approaches or investing more effort,” but he is “much more skeptical now than I was before that market forces are some sort of panacea, as they appear to be in some industries.” In the end, he thinks there will “always be a need for public accountability, if only to lower information costs to parents and students.”

I agree with much of this analysis. As I wrote at EdNext today:

There is no panacea. There is no perfect information just as there are no perfect bureaucrats or, for that matter, perfect parents. The question before us is how to design a system with imperfect people and imperfect information that will come as close as possible to providing every child with access to a high-quality education.

Getting parents the information they need to make good decisions is indeed a challenge. As Goodman notes, it is harder to identify a quality education than it is to identify a good meal or a good car. There is a great degree of subjectivity and the payoff is usually far in the future.

The question then becomes what sort of institutions are better equipped to address this challenge. As Lindsey Burke and I argue in a new report published by the Heritage Foundation and the Texas Public Policy Foundation, the technocratic approach of holding schools accountable on metrics that are easily observable and standardized creates perverse incentives that narrow the curriculum, stifle innovation, and can drive away quality schools from participating in the choice program. Instead, what we need are a variety of different third-party reviewers and platforms for parents and students to share their personal experiences in order to provide parents with the information they need.

I won’t rehash the entire argument here. You can either read the report on the short version at EdNext. The main point I wanted to highlight here was that the market is more likely than the government to overcome the information challenge — but for that to happen, there needs to be a large enough market. As I noted:

The K-12 education sector has historically lacked high-quality sources of information about school performance, but to a large extent that is because the vast majority of students attend their assigned district school. With little to no other educational options, there has been little parental need for information to compare competing options. And without much in the way of competition, existing private schools don’t feel great pressure to be forthcoming about performance data. However, as states implement educational choice policies, the demand for information will increase and schools that refuse to share their data will be at a competitive disadvantage. We are already seeing parents to turn organizations like GreatSchools.org and Niche.com to find information about schools they are considering and we should expect to see more organizations emerge as demand increases.

School choice programs that merely fill empty seats are not enough. They won’t bring the transformative change that Milton Friedman and others predicted. Modest choice programs produce modest results. It’s time to go bold.

Doug Harris has a new post that attempts to reply to the many critics of his New York Times op-ed, including me.  In the NYT piece Harris claimed that “one well-regarded study found that Detroit’s charter schools performed at about the same dismal level as its traditional public schools.” He also claimed that the relatively light regulatory approach to Detroit’s charter schools has led to “the biggest school reform disaster in the country.”  He prefers instead the heavy-regulation approach adopted in New Orleans, which he says has produced “impressive” results.  He then suggests that failing to believe these claims is “a triumph of ideology over evidence.”

Several critics, including me, noted that the “well-regarded study” Harris cites actually finds that Detroit charter schools are producing significantly greater gains than traditional public school alternatives — gains that are only slightly smaller than those in New Orleans and greater than in another high-regulation darling, Denver.  But Harris wants to continue posturing as the person backed by science and evidence, while describing his opponents as ideologues.  In his reply to critics in Education Next he uses the word “evidence” 16 times.

So, his defense of the claim that Detroit charters perform “at about the same dismal level as its traditional public schools” must be based on evidence, right? Actually, no.  He provides four arguments to rescue his assertion that “the failure of Detroit charter schools to improve student outcomes” is true despite the fact that the CREDO evidence he cited to make that claim shows otherwise.  Each one of those four arguments is based on assumptions, not evidence.  If you believe all of his assumptions, you might believe his claim that Detroit charter schools really are a disaster. But drawing conclusions that depend completely on assumptions and are contrary to the evidence he cites is certainly not science.  It seems more like a faith-based or ideological exercise.

Let’s consider his four arguments.  First he says, “given the lack of oversight in Detroit and evidence from other cities that some charter schools cherry-pick their preferred students, these results may make Detroit’s charter schools look better than they are.”  Got that?  He has no evidence that Detroit charters are cherry-picking students at all, let alone that they are doing so at a higher rate than in New Orleans, but he nevertheless posits that “if it’s happening, then the charter effects on achievement [in Detroit] are inflated.”

Second, given Harris’ assumed concerns about cherry-picking in Detroit charter schools and the inability of the CREDO study to account for that, he examines evidence from the urban NAEP test and finds that the city of Detroit has experienced below average growth in those scores in recent years.  Using NAEP results from the entire city to draw conclusions about Detroit’s charter schools requires a host of assumptions.  He’d have to assume that test results from all schools, charter and traditional, somehow speak to the effectiveness of charter schools.  He’d have to assume that demographic and other non-school factors in Detroit do not affect the comparison of test growth in Detroit relative to other cities.

A number of education analysts have coined a term — misNAEPery —  to capture how unreasonable it is to make the assumptions required to use NAEP to compare policies across jurisdictions.  Oddly, some of those analysts who like to accuse others of misNAEPery, like Morgan Polikoff and Matt Barnum, have somehow failed to denounce Harris’ use of misNAEPery in both the NYT op-ed and in the Ed Next reply to critics.  Both have even “retweeted” Harris’ new post, so we can assume they’ve read it.

And just to anticipate concerns about my own consistency on the use of NAEP, I think comparisons using NAEP that control for observed demographics are about as convincing as the CREDO results, which also rely on comparisons controlling for some observed characteristics.  Information from NAEP or CREDO can be interesting or suggestive, which is why I say that Arizona charters “appear” to be doing very well, even if I am not convinced that any of this is causal.  At the very least, it is useful to offer a disclaimer that neither NAEP nor CREDO provides convincing causal evidence, even if confession does not assure absolution.  Harris does not offer any disclaimer and instead uses his NAEP comparisons to bolster his assumption that Detroit charters may be cherry-picking.

Third, Harris builds on the observation that Detroit city has very low NAEP test scores to assume that “the extraordinarily low standing of the city as a whole, to the degree it is caused by low performance of traditional public schools, should make it easier to improve student outcomes when trying something new.”  It is also quite plausible — perhaps more plausible — that a city with extraordinarily low test scores also has severe social and economic problems that are outside of the control of schools, which would make it harder for charters to improve outcomes.

Lastly, Harris argues that even if Detroit charters have produced gains, the gains are smaller than those in New Orleans, so we should prefer the regulatory approach used in New Orleans to the one in Detroit. But this assumes that any greater gains produced by New Orleans charter schools are caused by the regulatory approach in that city.  In fact, we have no idea whether New Orleans’ regulations helped, hurt, or had no effect on how large the gains in that city were. For all we know, the gains made by New Orleans charters are largely attributable to the importation of top-notch human capital from elite colleges and a huge increase in per pupil spending, and that these gains were made despite the hindrance of burdensome regulations.  The heavy regulations in Denver somehow failed to produce gains as large as those in Detroit.

Other than these four-assumption-dependent arguments, Harris offers very little to defend the strong claims he made in the NYT that Detroit charter schools have been a “failure” and a “disaster.”  He does cite some national “charter-friendly” organizations, like CRPE and NACSA,  as being critical of Detroit charters.  But that is an argument from authority — not evidence.

He also falsely suggests that Detroit has failed to close failing charter schools.  In fact, 30% of Detroit’s charter schools have been shuttered.  And another national charter-friendly organization, NAPCS, gives Michigan higher marks on closure than Louisiana, noting that Michigan has closed more charter schools than Louisiana, 47 to 26.

Harris also cites two voucher studies with negative results — one of which is an RCT and the other not — as proof that lower regulation approaches are less effective.  Leaving aside the distinct possibility that the negative RCT result for Louisiana was actually a function of the over-regulation of that program, it’s important to note that Harris fails to cite the entire literature of rigorous studies on the effects of private school choice programs, which overwhelmingly shows positive outcomes.

I agree with Doug that ideology, which could more kindly be described as “principles” or “values,” has an important role to play in policymaking.  I just disagree with him that the evidence clearly shows the superiority of a high-regulation approach to school choice.  I think it’s more accurate to say that the evidence is unclear on this matter, which means that we may — appropriately — need to rely more on our values, principles, and broader ideology when deciding how to proceed.

[Edited to remove tangential comment that was poorly phrased.]

(Guest Post by Jason Bedrick)

Wise words from Rev. Alfred Cockfield II of Brooklyn:

Many members of our congregation live in neighborhoods where their local schools are failing. These parents care deeply about having more than one public school option because it means the difference between their child getting a quality education or just getting passed along.

President Obama and Secretary of Education John King have been strong proponents of public charter schools. They understand what parents in our congregation understand: that public charter schools and school choice are vital to closing the racial achievement gap.

And perhaps surprisingly, so does Donald Trump. For all of his rhetoric during the campaign, he is a strong supporter of public charter schools — and believes the status quo must change.

In the months ahead, we must remain vigilant, and be prepared to challenge the president-elect whenever he proposes a policy that threatens our communities. But we can’t let that stop the progress we know we can make and must make. And nowhere is the need to push forward together clearer than with charter schools.

If President-elect Trump is truly willing to support policies that give the highest-need children the school options that they desperately need, we should be willing to listen. That does not mean compromising our values — rather, it means authentically representing the needs of our communities, whether fighting back against a policy we disagree with or advocating for one we believe will help our children.

That’s the balancing act we face moving forward. And we cannot pretend it will be easy. But we owe it to our children to attempt to navigate the next four years with eyes wide open, and with their interests always front and center.

School choice is on the march.  Even before Trump’s election and his selection of Betsy DeVos as Secretary of Education, charter and private school choice have been steadily expanding every year.  Expanding school choice, like almost all of education reform, occurs in the states, so who is in charge in DC will not make too much of a difference other than turning a headwind into a tailwind.

But there is a certain crowd whose lives and thoughts revolve around what is happening in DC, so they have become all aflutter with either hope or dread about the prospects of a Trump presidency for school choice.  Those motivated by dread have fired first in the looming Regulation War, arguing that choice only works — if it works at all — within a heavy regulatory framework to ensure quality.  If choice can’t be stopped at least it can be controlled and that control can improve the choices parents will make.

The problem with the pro-regulation argument is that you can’t ensure quality if you don’t have the ability to predict quality.  It’s true that parents have a hard time anticipating how different schools might affect long-term outcomes and are quite likely to make mistakes in choosing schools.  And some emphasize this fact to justify a strong role for regulators in controlling the range and type of options from which parents can choose.  But what they rarely consider is whether regulators are any better at anticipating how schools will affect long-term outcomes.  After all, regulators can’t protect families against making mistakes if they are equally or more likely to make those mistakes themselves.

I’ve written previously about the disconnect between near-term test score gains and changes in later life outcomes.  If we can’t reliably use rigorously identified test score gains to predict later life outcomes, then on what basis will regulators be able to judge quality to protect families against making bad choices?  And the situation is even worse because most regulators making decisions about what choice schools should be opened, expanded, or closed are not relying on rigorously identified gains in test scores — they just look primarily at the levels of test scores and call those with low scores bad.  Of course, all this does is punish and discourage schools from attempting to serve the most challenging students since the level of scores is more a reflection of student background than school quality.

But let’s say you don’t believe me about the weak predictive power of test score gains and are determined to use tests as the main indicator of school quality.  We are still left with the question of whether regulators are any good at identifying which schools will contribute to test score gains.  Fortunately, we have a recent study that examined whether the criteria used by regulators in New Orleans are predictive of test score growth — even if we accept test gains as a reliable indicator of quality.  The bottom line is that none of the factors used by authorizers to open or renew charter schools in New Orleans were predictive of how much test score growth these schools could produce later on.

In particular, the study examines ratings derived from criteria favored by the National Association of Charter School Authorizers (NACSA) to see if they are predictive of test score growth or enrollment growth.  Neither the NACSA aggregate rating nor individual factors, like school governance features, educational plans, or financial and enrollment characteristics were predictive.  As the study concluded, “None of the factors, however, predict the main dependent variables of interest: SPS, value-added, enrollment, and enrollment growth, as direct measures of future school performance. ”

If regulators are unable to predict which schools will be good (assuming, falsely, that test score gains are a reliable indicator of good schools), how are they supposed to “protect” parents from making bad choices about schools?  The case for regulation is often expressed as a tautology: people will make better choices if they only have good schools from which to choose. But this presupposes that regulators can identify and eliminate bad choices.  And if regulators really could do this, why have school choice at all?  Why not just ensure that every student succeeds by forbidding any school from being bad?

No one should be surprised that NACSA’s criteria have no relationship to their own metric for school quality — test score growth — given how well Arizona charter schools appear to be doing even while NACSA gives the state a very low score for charter quality.  What might be more surprising is that the recent study I’ve been referencing that finds no connection between the criteria regulators use and future test score growth was co-authored by none other than Doug Harris.  Yes, that’s the same Doug Harris who wrote the NYT op-ed claiming that the lack of regulation in Detroit contributed to “the biggest school reform disaster in the country,” while New Orleans’ more heavily regulated approach produced “impressive” results.

As Ramesh Ponnuru has noted, using the same CREDO research that Doug cites, Detroit charter schools produce significantly better outcomes than the traditional public school alternatives.  In fact, the test score gains achieved by Detroit charters exceed those in heavily regulated Denver and only slightly lag those in New Orleans.  (See the table above for a graphic presentation of results across cities).

Mind you, I do not believe the CREDO results because that research only matches on observed characteristics and therefore does not have a rigorous identification of causal effects.  But Doug seems to believe that research even though it undermines the very argument he was making.  And one can only assume that Doug believes his own research showing that regulators lack effective tools to identify and predict good schools.  Doug accuses DeVos of being guilty of “a triumph of ideology over evidence,” but it seems like there is a lot of that going around.