You Can’t Regulate Quality If You Can’t Predict Quality

School choice is on the march.  Even before Trump’s election and his selection of Betsy DeVos as Secretary of Education, charter and private school choice have been steadily expanding every year.  Expanding school choice, like almost all of education reform, occurs in the states, so who is in charge in DC will not make too much of a difference other than turning a headwind into a tailwind.

But there is a certain crowd whose lives and thoughts revolve around what is happening in DC, so they have become all aflutter with either hope or dread about the prospects of a Trump presidency for school choice.  Those motivated by dread have fired first in the looming Regulation War, arguing that choice only works — if it works at all — within a heavy regulatory framework to ensure quality.  If choice can’t be stopped at least it can be controlled and that control can improve the choices parents will make.

The problem with the pro-regulation argument is that you can’t ensure quality if you don’t have the ability to predict quality.  It’s true that parents have a hard time anticipating how different schools might affect long-term outcomes and are quite likely to make mistakes in choosing schools.  And some emphasize this fact to justify a strong role for regulators in controlling the range and type of options from which parents can choose.  But what they rarely consider is whether regulators are any better at anticipating how schools will affect long-term outcomes.  After all, regulators can’t protect families against making mistakes if they are equally or more likely to make those mistakes themselves.

I’ve written previously about the disconnect between near-term test score gains and changes in later life outcomes.  If we can’t reliably use rigorously identified test score gains to predict later life outcomes, then on what basis will regulators be able to judge quality to protect families against making bad choices?  And the situation is even worse because most regulators making decisions about what choice schools should be opened, expanded, or closed are not relying on rigorously identified gains in test scores — they just look primarily at the levels of test scores and call those with low scores bad.  Of course, all this does is punish and discourage schools from attempting to serve the most challenging students since the level of scores is more a reflection of student background than school quality.

But let’s say you don’t believe me about the weak predictive power of test score gains and are determined to use tests as the main indicator of school quality.  We are still left with the question of whether regulators are any good at identifying which schools will contribute to test score gains.  Fortunately, we have a recent study that examined whether the criteria used by regulators in New Orleans are predictive of test score growth — even if we accept test gains as a reliable indicator of quality.  The bottom line is that none of the factors used by authorizers to open or renew charter schools in New Orleans were predictive of how much test score growth these schools could produce later on.

In particular, the study examines ratings derived from criteria favored by the National Association of Charter School Authorizers (NACSA) to see if they are predictive of test score growth or enrollment growth.  Neither the NACSA aggregate rating nor individual factors, like school governance features, educational plans, or financial and enrollment characteristics were predictive.  As the study concluded, “None of the factors, however, predict the main dependent variables of interest: SPS, value-added, enrollment, and enrollment growth, as direct measures of future school performance. ”

If regulators are unable to predict which schools will be good (assuming, falsely, that test score gains are a reliable indicator of good schools), how are they supposed to “protect” parents from making bad choices about schools?  The case for regulation is often expressed as a tautology: people will make better choices if they only have good schools from which to choose. But this presupposes that regulators can identify and eliminate bad choices.  And if regulators really could do this, why have school choice at all?  Why not just ensure that every student succeeds by forbidding any school from being bad?

No one should be surprised that NACSA’s criteria have no relationship to their own metric for school quality — test score growth — given how well Arizona charter schools appear to be doing even while NACSA gives the state a very low score for charter quality.  What might be more surprising is that the recent study I’ve been referencing that finds no connection between the criteria regulators use and future test score growth was co-authored by none other than Doug Harris.  Yes, that’s the same Doug Harris who wrote the NYT op-ed claiming that the lack of regulation in Detroit contributed to “the biggest school reform disaster in the country,” while New Orleans’ more heavily regulated approach produced “impressive” results.

As Ramesh Ponnuru has noted, using the same CREDO research that Doug cites, Detroit charter schools produce significantly better outcomes than the traditional public school alternatives.  In fact, the test score gains achieved by Detroit charters exceed those in heavily regulated Denver and only slightly lag those in New Orleans.  (See the table above for a graphic presentation of results across cities).

Mind you, I do not believe the CREDO results because that research only matches on observed characteristics and therefore does not have a rigorous identification of causal effects.  But Doug seems to believe that research even though it undermines the very argument he was making.  And one can only assume that Doug believes his own research showing that regulators lack effective tools to identify and predict good schools.  Doug accuses DeVos of being guilty of “a triumph of ideology over evidence,” but it seems like there is a lot of that going around.


5 Responses to You Can’t Regulate Quality If You Can’t Predict Quality

  1. Greg Forster says:

    “It seems like there is a lot of that going around.”

    Psychologists call it “projection.”

  2. Zeev Wurman says:

    There seem to be two ways to interpret your underlying argument.

    (a) That there exist some — still hidden or unknown — characteristics (and measures) of school effectiveness, and they just wait to be discovered, or

    (b) There are no such things as broad characteristics of school effectiveness because each and every child has somewhat different needs (whether innate or family/society engendered) and hence every measure we may come up with will have — on the *average* — little or no discernible effects.

    [Note: The way I understand it, (b) above doesn’t necessarily imply that there are no strong and broad *negative* correlates with school effectiveness such as hunger, or torture, or absence of teacher’s subject-matter knowledge.]

    While I originally started as a believer in (a), more and more I lean towards (b) as time passes. What it means, to me, is that there may be multiple characteristics of effective schooling but each is effective only to a subset of students. If a student is placed in an environment not fitting with his or her character it will rarely work well, while it may work very well for some others.

    The consequence of this, as I see it, is that individual monitoring of students, and intervention in case of mismatch, are of cardinal importance. Who can provide better individual monitoring than student’s parents, and what can offer better intervention — on the average — than the availability of school choice?

    That is my current argument for school choice.

    • Greg Forster says:

      Doesn’t it pretty much have to be a combination? There can’t be no factors that lead to good education in most cases.

      • Zeev Wurman says:

        Clearly some practices/characteristics are helpful. This is implied by the “note” on (b). But what I am trying to say that there may not be *strong* and *universal* characteristics as implied by (a).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s