(This is an update of a post I wrote on August 25, 2008. It now includes the new Milwaukee study.)
In an earlier post I listed all analyses of the effects of U.S. vouchers on program participants using random-assignment experiments. Those studies tell us about what happens to the academic achievement of students who receive vouchers. But we all recognize that expanding choice and competition with vouchers may also have significant effects on students who remain in traditional public schools. Here is a brief summary of the research on that question.
In general, the evidence on systemic effects (how expanding choice and competition affects the performance of traditional public schools) has more methodological limitations than participant effects studies. We haven’t been able to randomly assign school districts to increased competition, so we have more serious problems with drawing causal inferences. Even devising accurate measures of the extent of competition has been problematic. That being said, the findings on systemic effects, like on participant effects, is generally positive and almost never negative.
Even in the absence of choice programs traditional public schools are exposed to some amount of competition. They may compete with public schools in other districts or with nearby private schools. A relatively large number of studies have examined this naturally occurring variation in competition. To avoid being accused of cherry-picking this evidence I’ll rely on the review of that literature conducted by Henry Levin and Clive Belfield. Here is the abstract of their review, in full:
“This article systematically reviews U.S. evidence from cross-sectional research on educational outcomes when schools must compete with each other. Competition typically is measured by using either the HerfindahlIndex or the enrollment rate at an alternative school choice. Outcomes are academic test scores, graduation/attainment, expenditures/efficiency, teacher quality, students’ post-school wages, and local housing prices. The sampling strategy identified more than 41 relevant empiricalstudies. A sizable majority report beneficial effects of competition, and many report statistically significant correlations. For each study, the effect size of an increase of competition by one standard deviation is reported. The positive gains from competition are modest in scope with respect to realistic changes in levels of competition. The review also notes several methodological challenges and recommends caution in reasoning from point estimates to public policy.”
There have also been a number of studies that have examined the effect of expanding competition or the threat of competition on public schools from voucher programs in Milwaukee and Florida. Here are all of the major studies of systemic effects of which I am aware from voucher programs in the US:
Milwaukee
Martin Carnoy, et al “Vouchers and Public School Performance,” Economic Policy Institute, October 2007;
Rajashri Chakrabarti, “Can Increasing Private School Participation and Monetary Loss in a Voucher Program Affect Public School Performance? Evidence from Milwaukee,” Federal Reserve Bank of New York, 2007; (forthcoming in the Journal of Public Economics)
Caroline Minter Hoxby, “The Rising Tide,” Education Next, Winter 2001;
Jay P. Greene and Ryan H. Marsh, “The Effect of Milwaukee’s Parental Choice Program on Student Achievement in Milwaukee Public Schools,” School Choice Demonstration Project Report, March 2009.
Florida
Federal Reserve Bank of New York Staff Report, Number 306, October 2007;
Jay P. Greene and Marcus A. Winters, “Competition Passes the Test,” Education Next, Summer 2004;
Cecilia Elena Rouse, Jane Hannaway, Dan Goldhaber, and David Figlio, “Feeling the Heat: How Low Performing Schools Respond to Voucher and Accountability Pressure,” CALDER Working Paper 13, Urban Institute, November 2007;
Martin West and Paul Peterson, “The Efficacy of Choice Threats Within School Accountability Systems,” Harvard PEPG Working Paper 05-01, March 23, 2005; (subsequently published in The Economic Journal, March, 2006)
Jay P. Greene and Marcus A. Winters, “The Effect of Special Education Vouchers on Public School Achievement: Evidence From Florida’s McKay Scholarship Program” Manhattan Institute, Civic Report Number 52, April 2008. (looks only at voucher program for disabled students)
Every one of these 9 studies finds positive systemic effects. It is importantto note that Rouse, et al are ambiguous as to whether they attribute the improvements observed to competition or to the stigma of Florida’s accountability system. The other three Florida studies perform analyses that support the conclusion that the gains were from competitive pressure rather than simply from stigma.
Also Carnoy, et al confirm Chakrabarti’s finding that Milwaukee public schools improved as the voucher program expanded, but they emphasize that those gains did not continue to increase as the program expanded further (nor did those gains disappear). They find this lack of continued improvement worrisome and believe that it undermines confidence one could have in the initial positive reaction from competition that they and others have observed. This and other analyses using different measures of competition with null results lead them to conclude that overall there is a null effect — even though they do confirm Chakrabarti’s finding of a positive effect.
I would also add that Greg Forster and I have a study of systemic effects in Milwaukee and Greg has a new study of systemic effects from the voucher program in Ohio. And Greg also has a neat study that shows that schools previously threatened with voucher competition slipped after Florida’s Supreme Court struck down the voucher provision. All of these studies also show positive systemic effects, but since they have not undergone external review and since I do not want to overstate the evidence, I’ve left them out of the above list of studies. People who, after reading them, have confidence in these three studies should add them to the list of studies on systemic effects.
The bottom line is that none of the studies of systemic effects from voucher programs finds negative effects on student achievement in public schools from voucher competition. The bulk of the evidence, both from studies of voucher programs and from variation in existing competition among public schools, supports the conclusion that expanding competition improves student achievement.
(edited to add study by Greg on post-voucher FL and Jay’s study on McKay vouchers for disabled students)
(Updated 4/27/09 to include the new Milwaukee study)
Posted by Jay P. Greene 
Jay P. Greene's Blog 