Surprise, Surprise, Surprise!

(Guest post by Patrick J. Wolf)

I’ve led or assisted with seven rigorous longitudinal evaluations of privately- or publicly-funded private school choice programs. Each one has yielded a big surprise.

The three-city evaluation Paul Peterson led discovered that partial-tuition K-12 scholarships had no clear effect on student test scores overall, but clearly benefited African American students.  The original evaluation of the DC Opportunity Scholarship Program that I led for the U.S. Department of Education found only suggestive evidence of achievement effects of the federal school voucher program, and only in reading, but identified big positive effects of the program on high school graduation rates. The Milwaukee Parental Choice Program evaluation I co-led with John Witte confirmed that the original urban school voucher program had clear effects on multiple measures of student educational attainment but also produced reading test score gains when the tests were high-stakes for the private schools in the program. An experimental study of a partial-tuition scholarship program in India, for which I provided assistance, found clear achievement effects only for girls.

Then there is our four-year longitudinal evaluation of the Louisiana Scholarship Program. In our first set of reports we found that the LSP had positive effects on public school racial integration, as reported here, here, and here. We also discovered that competitive pressure from the statewide expansion of the school voucher program in 2012 had neutral or positive effects on the test scores of students in affected public schools. Our big surprise, however, was that participants in the voucher program scored significantly lower on the state accountability test than their control group peers, especially in the first year of the program and particularly in math. Those negative test score effects of the voucher program were somewhat smaller after two years and even statistically insignificant in Year 3, when the state switched to a different test and held schools harmless for the results.  What would happen in Year 4?  More surprises, it turns out.

Today we released the results of our final set of four technical research reports on the LSP.  The experimental impacts of winning a lottery to your first-choice private school and enrolling in that school for any period of time were negative and back to statistically significant for all of our statistical models in math and some of them in reading. African American students experienced smaller negative achievement effects than did students of other races. Students whose first-choice private schools had higher tuitions, larger enrollments, and longer school days experienced relatively “better” test score effects than students whose first-choice schools didn’t have those features.

Winning an LSP school lottery had no impact on the rates at which students in our study subsequently enrolled in either a two-year or four-year college. The rate was 60.0% for students in the LSP experimental treatment group and 59.5% in the experimental control group. Those college enrollment rates are relatively high for the population of low-income students eligible to apply for the LSP, in part because Louisiana has enacted a series of programs to encourage college access.

Finally, the LSP succeeded in attracting students from economically and socially disadvantaged backgrounds. The students who remained in the LSP three years after applying to it were more likely to be eligible for the federal lunch program, African American, and female than the average K-12 student in Louisiana. Students with lower initial test scores than the state average more likely to apply to the LSP than were students with higher test scores.  Among the students who won LSP school lotteries, those with lower initial test scores were more likely to use an LSP voucher for at least three years.

Debates will rage about what has been learned from this one, latest, rigorous evaluation of a private school choice program. Commentators should keep in mind that the LSP has design features that make it unlike most school choice programs. It is the only statewide school voucher program that requires participating schools to adopt an open admissions policy and administer the state accountability test to their voucher-using students. Survey experiments here, and here, have found that those two regulatory provisions tend to decrease the interest of private school leaders in participating in school choice programs. Only about one-third of Louisiana private schools choose to participate in the LSP.

Serious, rigorous studies of private school choice programs should continue. So far, results keep surprising because it appears that the effects of these programs are highly dependent on their design and context. I wouldn’t be surprised if more surprises await.

6 Responses to Surprise, Surprise, Surprise!

  1. George Mitchell says:

    Rare is the school choice program that could meet the test of being “well designed.” Compromises, often with well organized opponents of choice, mean programs are freighted with financial and regulatory burdens and limits. One thing that’s not surprising: Pat Wolf and colleagues continue to explore and honestly present findings.

    • Well said, George. The only extra burden with the design of programs is that it has been ed reformers themselves who have championed many of these counter-productive regulations. It wasn’t opponents who championed these deign flaws, it was the biggest supporters.

      • George Mitchell says:

        So-called “education reformers.” I have lost track of the meetings where “our side” id’d the concessions they wanted to make right out of the gate to get “buy-in.” I remember WAY BACK when Paul Peterson pointed out that the Milwaukee test results were especially notable given the limitations of the program design.

  2. matthewladner says:

    So having 2/3 of schools that could participate shun your program is bad for student outcomes.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s