Jay P. Greene's Blog

School choice is on the march.  Even before Trump’s election and his selection of Betsy DeVos as Secretary of Education, charter and private school choice have been steadily expanding every year.  Expanding school choice, like almost all of education reform, occurs in the states, so who is in charge in DC will not make too much of a difference other than turning a headwind into a tailwind.

But there is a certain crowd whose lives and thoughts revolve around what is happening in DC, so they have become all aflutter with either hope or dread about the prospects of a Trump presidency for school choice.  Those motivated by dread have fired first in the looming Regulation War, arguing that choice only works — if it works at all — within a heavy regulatory framework to ensure quality.  If choice can’t be stopped at least it can be controlled and that control can improve the choices parents will make.

The problem with the pro-regulation argument is that you can’t ensure quality if you don’t have the ability to predict quality.  It’s true that parents have a hard time anticipating how different schools might affect long-term outcomes and are quite likely to make mistakes in choosing schools.  And some emphasize this fact to justify a strong role for regulators in controlling the range and type of options from which parents can choose.  But what they rarely consider is whether regulators are any better at anticipating how schools will affect long-term outcomes.  After all, regulators can’t protect families against making mistakes if they are equally or more likely to make those mistakes themselves.

I’ve written previously about the disconnect between near-term test score gains and changes in later life outcomes.  If we can’t reliably use rigorously identified test score gains to predict later life outcomes, then on what basis will regulators be able to judge quality to protect families against making bad choices?  And the situation is even worse because most regulators making decisions about what choice schools should be opened, expanded, or closed are not relying on rigorously identified gains in test scores — they just look primarily at the levels of test scores and call those with low scores bad.  Of course, all this does is punish and discourage schools from attempting to serve the most challenging students since the level of scores is more a reflection of student background than school quality.

But let’s say you don’t believe me about the weak predictive power of test score gains and are determined to use tests as the main indicator of school quality.  We are still left with the question of whether regulators are any good at identifying which schools will contribute to test score gains.  Fortunately, we have a recent study that examined whether the criteria used by regulators in New Orleans are predictive of test score growth — even if we accept test gains as a reliable indicator of quality.  The bottom line is that none of the factors used by authorizers to open or renew charter schools in New Orleans were predictive of how much test score growth these schools could produce later on.

In particular, the study examines ratings derived from criteria favored by the National Association of Charter School Authorizers (NACSA) to see if they are predictive of test score growth or enrollment growth.  Neither the NACSA aggregate rating nor individual factors, like school governance features, educational plans, or financial and enrollment characteristics were predictive.  As the study concluded, “None of the factors, however, predict the main dependent variables of interest: SPS, value-added, enrollment, and enrollment growth, as direct measures of future school performance. ”

If regulators are unable to predict which schools will be good (assuming, falsely, that test score gains are a reliable indicator of good schools), how are they supposed to “protect” parents from making bad choices about schools?  The case for regulation is often expressed as a tautology: people will make better choices if they only have good schools from which to choose. But this presupposes that regulators can identify and eliminate bad choices.  And if regulators really could do this, why have school choice at all?  Why not just ensure that every student succeeds by forbidding any school from being bad?

No one should be surprised that NACSA’s criteria have no relationship to their own metric for school quality — test score growth — given how well Arizona charter schools appear to be doing even while NACSA gives the state a very low score for charter quality.  What might be more surprising is that the recent study I’ve been referencing that finds no connection between the criteria regulators use and future test score growth was co-authored by none other than Doug Harris.  Yes, that’s the same Doug Harris who wrote the NYT op-ed claiming that the lack of regulation in Detroit contributed to “the biggest school reform disaster in the country,” while New Orleans’ more heavily regulated approach produced “impressive” results.

As Ramesh Ponnuru has noted, using the same CREDO research that Doug cites, Detroit charter schools produce significantly better outcomes than the traditional public school alternatives.  In fact, the test score gains achieved by Detroit charters exceed those in heavily regulated Denver and only slightly lag those in New Orleans.  (See the table above for a graphic presentation of results across cities).

Mind you, I do not believe the CREDO results because that research only matches on observed characteristics and therefore does not have a rigorous identification of causal effects.  But Doug seems to believe that research even though it undermines the very argument he was making.  And one can only assume that Doug believes his own research showing that regulators lack effective tools to identify and predict good schools.  Doug accuses DeVos of being guilty of “a triumph of ideology over evidence,” but it seems like there is a lot of that going around.

Over the last few years I have developed a deeper skepticism about the reliability of relying on test scores for accountability purposes.  I think tests have very limited potential in guiding distant policymakers, regulators, portfolio managers, foundation officials, and other policy elites in identifying with confidence which schools are good or bad, ought to be opened, expanded, or closed, and which programs are working or failing.  The problem, as I’ve pointed out in several pieces now, is that in using tests for these purposes we are assuming that if we can change test scores, we will change later outcomes in life.  We don’t really care about test scores per se, we care about them because we think they are near-term proxies for later life outcomes that we really do care about — like graduating from high school, going to college, getting a job, earning a good living, staying out of jail, etc…

But what if changing test scores does not regularly correspond with changing life outcomes?  What if schools can do things to change scores without actually changing lives?  What evidence do we actually have to support the assumption that changing test scores is a reliable indicator of changing later life outcomes?

This concern is similar to issues that have arisen in other fields about the reliability of near-term indicators as proxies for later life outcomes.  For example, as one of my colleagues noted to me, there are medicines that are able to lower cholesterol levels but do not reduce — or even may increase — mortality from heart disease.  It’s important that we think carefully about whether we are making the same type of mistake in education.

If increasing test scores is a good indicator of improving later life outcomes, we should see roughly the same direction and magnitude in changes of scores and later outcomes in most rigorously identified studies.  We do not.  I’m not saying we never see a connection between changing test scores and changing later life outcomes (e.g. Chetty, et al); I’m just saying that we do not regularly see that relationship.  For an indicator to be reliable, it should yield accurate predictions nearly all, or at least most, of the time.

To illustrate the un-reliability of test score changes, I’m going to focus on rigorously identified research on school choice programs where we have later life outcomes.  We could find plenty of examples of disconnect from other policy interventions, such as pre-school programs, but I am focusing on school choice because I know this literature best.  The fact that we can find a disconnect between test score changes and later life outcomes in any literature, let alone in several, should undermine our confidence in test scores as a reliable indicator.

I should also emphasize that by looking at rigorous research I am rigging things in favor of test scores.  If we explored the most common use of test scores — examining the level of proficiency — there are no credible researchers who believe that is a reliable indicator of school or program quality.  Even measures of growth in test scores or VAM are not rigorously identified indicators of school or program quality as they do not reveal what the growth would have been in the absence of that school or program.  So, I think almost every credible researcher would agree that the vast majority of ways in which test scores are used by policymakers, regulators, portfolio managers, foundation officials, and other policy elites cannot be reliable indicators of the ability of schools or programs to improve later life outcomes.

With the evidence below I am exploring the largely imaginary scenario in which test scores changes can be attributed to schools or programs with confidence.  Even then, the direction and magnitude of changing test scores does not regularly correspond with changing later life outcomes.  I’ve identified 10 rigorously designed studies of charter and private school choice programs with later life outcomes.  I’ve listed them below with a brief description of their findings and hyperlinks so you can read the results for yourself.

Notice any patterns? Other than the general disconnect between test scores and later life outcomes (in both directions), I notice that the No Excuses charter model that is currently the darling of the ed reform movement and that New York Times columnists have declared as the only type of “Schools that Work” tend not to fare nearly as well in later outcomes as they do on test scores.  Meanwhile the unfashionable private choice schools and Mom and Pop charters seem to do much better on later life outcomes than at changing test scores.  I don’t highlight this pattern as proof that we should shy away from No Excuses charters.  I only mention it to suggest ways in which over-relying on test scores and declaring with confidence that we know what works and what doesn’t can lead to big policy mistakes.

Here are the 10 studies:

1. Boston charters (Angrist, et al, 2014) – Huge test score gains, no increase in HS grad rate or postsecondary attendance. Shift from 2 to 4 yr
2. Harlem Promise Academy (Dobbie and Fryer, 2014) – Same as Boston charters
3. KIPP (Tuttle, et al, 2015) – Large test score gains, no or small effect on HS grad rate, depending on analysis used
4. High Tech High (Beauregard, 2015) – Widely praised for improving test scores, no increase in college enrollment
5. SEED Boarding Charter (Unterman, et al, 2016) – same as Boston charters
6. TX No Excuses charters (Dobbie and Fryer, 2016) – Increase test scores and college enrollment, but no effect on earnings
7. Florida charters (Booker, et al, 2014) – No test score gains but large increase in HS grad rate, college attendance, and earnings
8. DC vouchers (Wolf, et al, 2013) – Little or no test score gain but large increase in HS grad rate
9. Milwaukee vouchers (Cowen, et al, 2013) – same as DC
10. New York vouchers (Chingos and Peterson, 2013) – modest test score gain, larger college enrollment improvement