Let’s Search for Sweet Spots, but with modesty please

(Guest Post by Matthew Ladner)

I have a number of friends who have either helped develop or have signed on to a Statement of Principles regarding a three sector reform strategy and what they view as a desirable level of state oversight of private choice programs.  This post will work better for you if you go and read the document first.

The needle starts to scratch across the vinyl for me at:

Even with the expanded choice to the private sector, they also have produced modest results.

This has become a mantra in recent years, but I believe that this statement reflects an incomplete understanding of the research results, and specifically a lack of understanding regarding our random assignment studies of voucher programs. The basic takeaway from the random assignment studies in my view is as follows: the test score impacts are modest but often statistically significant within the three year window that we can reliably study them.

So the Milwaukee Parental Choice Program offered $6,400 vouchers to very low-income inner city parents whose other options were to attend a district spending $14,000 per child and/or charter schools spending somewhere between the voucher and district spending.  We have several random assignment studies of the test score impacts that find that the experimental group basically stays on grade level (a triumph for poor inner city children) whereas the control group declines year by year.  You get to watch this process for about three years before the random assignment breaks down on you.

What happens to test scores after Year 3?  No one knows for sure- these studies fall apart over time.  We do know things however about what happens regarding high-school graduation, college attendance, college persistence, etc.  Borrowing a slide that Pat Wolf presented at the Alliance for School Choice conference:

Slide11

So basically you are less likely to graduate in 5 years (first red column) because you are more likely to graduate on time, less likely to graduate from a two-year college (second red) because you are more likely to be going to a four-year college.  The blue columns are all positive impacts from having been a choice student.

Now if you are determined to cling to the “modest” camp by saying that you wish these impacts were even larger, well, I do too.  I also wish that Chuck Norris’ tears really did cure cancer.  At this point it might be appropriate to raise the question as to just how much a positive impact we should reasonably expect from a program giving profoundly disadvantaged children a $6,400 coupon.  Although we don’t know what happens after a few years of random assignment study, those graduation figures ultimately are far more important than 6th grade math scores.

Being far more likely to graduate from high-school and college for less than half the money sounds like a triumph to me, albeit one that we could and should hope to improve upon through more robust program designs.  The standard here should not be to expect MPCP to transform every last profoundly disadvantaged inner city child into a Dean’s List Ivy Leaguers.  Rather in judging the impact of MPCP we should look at it on a return per dollar invested basis.  When you look at it appropriately through this ROI it is clear that the return on MPCP has been quite good, and that we should be looking for ways to get even more of it.

Then I got to this statement:

We know that smart accountability measures can ensure that public money and young lives are not invested in low-performing private schools.

The statement offers no evidence to support this claim, and moreover the claim itself dodges the more important question of costs and benefits to regulation.  Is it possible for “smart” accountability to keep young lives out of low-performing private schools?  Sure it’s possible.  Smart training can ensure that I could go from being a 46-year-old policy wonk to heavyweight champion of the world. I mean it is possible right? Is it also possible, even highly likely, for the whole enterprise to go south on you in a variety of different ways? Yep, that’s very possible too.

Who is going to administer these smart accountability measures and who will administer them a few years later?  What about 25 years from now?  How often will these people do something they think is smart which proves to be otherwise?  Unless we want to have the Federal Reserve administer these programs, how long will it be until politics will subvert the process of “smart” technocratic policymaking?  Also like the Fed, the costs of technocratic mistakes may prove quite costly.

Even well-intentioned efforts at “smart accountability” could easily backfire.  Let’s take Louisiana as an example.  Louisiana policymakers decided to grade all their schools A-F based upon a state accountability test tied to the state academic standards, and then decided to create a mechanism to remove low-performing schools from eligibility to take new students.  This probably sounds clever at a Georgetown cocktail party, but in Louisiana two-thirds of the state’s private schools have decided to stay out of the program, denying thousands of seats to low-income children attending relatively poor performing public schools in one of the lowest performing states in the union.

Ooops.

Let’s take things a step further. Is it possible that the one-third of Louisiana private schools that chose to participate in the program may have had a selection bias towards being more on the financially desperate side than those that have decided to stay out?  I have no data to support that this in fact did happen, but who would be surprised if it in fact did happen?  The correlation between financial desperation and academic ineptitude often proves strong.  In such a case the initial impact of the regulatory regime might have precisely the opposite of what was intended with many higher performing schools choosing to keep their distance.  Worse still, it might create an incentive for private schools to engage in the same sort of gaming strategies that have been common in states with rising state test scores but flat NAEP scores- teaching to test items rather than to standards (Arizona is waving hello!).  Finally of course it is no triumph if the schools do actually teach the state standards because the whole idea of a choice program is to provide, well, meaningfully varying choices for parents.  If you want state tests and standards in Louisiana you already have thousands of options available to you in the form of district and charter schools.

In the end of the day, policymakers must make decisions about where to draw the line in such matters. We have no wrong or right answers here, only preferences. Personally I believe that choice programs should provide academic transparency to the public in ways designed to have the lightest possible touch on the curricular independence of schools.  I’m willing to sacrifice some level of private school participation in return for transparency.  Preferences will vary and we will learn things along the way through variation between programs.  What I think I have learned however is that Arizona’s transparency-light programs represent a costly obstacle to building broad support, and that the Louisiana and Indiana model has far too many private schools saying “thanks but no thanks.”

To my friends who crafted and signed on to this statement I say only that we should continue the dialogue and gather more information.  I don’t believe in regulation free programs nor do I expect or desire for us to pass any, so I agree with you to a degree. I however strongly suspect that many of you are underestimating the cost of regulation and overestimating the capacity of technocratic regimes.

 

 

 

 

 

 

 

 

Advertisements

3 Responses to Let’s Search for Sweet Spots, but with modesty please

  1. Greg Forster says:

    Would smart accountability keep vouchers away from low-performing schools? Absolutely it would – if we define “low-performing schools” as “schools the powerful don’t like.”

    If school choice regulations get tighter rather than looser, e.g. by imposition of Common Core tests on participating schools, I foresee the emergence of a cartel. Rather than a monopoly on schooling by Big Government, you’ll have an oligopoly on schooling by Big Government and Big Private School (Catholic schools and other large, complacent, bureaucratic systems). They’ll work together to suppress innovative upstarts. Available choices will go from “mediocre government schools” to “mediocre government schools or very-slightly-less-mediocre private schools.”

    Serious improvement in education is going to come from innovative educational entrepreneurs who aren’t interested in building carbon copies of the existing system.

  2. matthewladner says:

    There is an air of the generals fighting the last war about, giving the steady transformation of what constitutes a “school” but having said that I do believe that provisions for academic transparency should be made when practical.

  3. I’m a circle with hands pointing to a twelve and
    a six; their position changes with minutely ticks – clock’s face.
    Just follow the path as this level hass little to explore since it is inside a pipe
    factory. You can probably remove them woth a map editor, but that’snot my area of expertise.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s