Pat Wolf In Ed Next

August 20, 2009

Pat Wolf has an article summarizing and clarifying the latest evidence from the official evaluation of the D.C. voucher program newly posted at Education Next.

The part that struck me the most was how strong the DC voucher results are compared to the results of all of the other rigorous evaluations sponsored by the U.S. Department of Education:

The achievement results from the D.C. voucher evaluation are also striking when compared to the results from other experimental evaluations of education policies. The National Center for Education Evaluation and Regional Assistance (NCEE) at the IES has sponsored and overseen 11 studies that are RCTs, including the OSP evaluation. Only 3 of the 11 education interventions tested, when subjected to such a rigorous evaluation, have demonstrated statistically significant achievement impacts overall in either reading or math. The reading impact of the D.C. voucher program is the largest achievement impact yet reported in an RCT evaluation overseen by the NCEE. A second program was found to increase reading outcomes by about 40 percent less than the reading gain from the DC OSP. The third intervention was reported to have boosted math achievement by less than half the amount of the reading gain from the D.C. voucher program. Of the remaining eight NCEE-sponsored RCTs, six of them found no statistically significant achievement impacts overall and the other two showed a mix of no impacts and actual achievement losses from their programs.

1 Comment | vouchers | Tagged: , , | Permalink
Posted by Jay P. Greene

Mistaken AJC Voucher Editorial Held Accountable

February 9, 2009

 

One of the great things about these here inter-web thingies is their ability to hold newspapers accountable when they make mistakes.  And the editorial by Maureen Downey that the Atlanta Journal Constitution ran last week on vouchers was very much mistaken.  In it Downey  claimed “in the handful of states that have conducted experiments with vouchers, the results contradict claims of improvement by Johnson and other voucher advocates… Yet, in return for zero impact, Johnson proposes to dismantle public education in Georgia.”  She also described “vouchers as a threat to the bedrock American belief that public education is critical to the health of the democracy and should not be sacrificed to political agendas.” 

To support her overwrought claims she cites a newspaper article on Ohio’s voucher program, studies of the voucher programs in DC and Milwaukee conducted by my colleague Pat  Wolf, and a review of the literature by Barrow and Rouse.  Unfortunately she cites all of them selectively or misinterprets their findings as showing “zero impact.”  Fortunately, Pat Wolf noticed her incorrect interpretation of his work and sent a letter, which the AJC ran today.

But letters are limited in length and less salient than the editorials they attempt to correct.  In the old days when newspapers were the only game in town, it was very difficult to hold newspapers accountable for editorials that were factually inaccurate.  They might have run letters, like the one Pat Wolf submitted, but they wouldn’t even have to do that if they didn’t want to.

With the inter-webs we not only have Pat Wolf’s letter in the AJC, we can also circulate it by posting it on blogs, like I just did.  And we can add additional material, for which there would have been no space in the letters section.  So let me add that here is a complete list of random-assignment studies of the effects of vouchers on students who use them.  Here is a summary of the effect of vouchers on the public school system.  And here is random-assignment research on the effect of charter schools on participants.  And if she thinks choice destroys democracy, here is a review of that literature showing that she is mistaken about that as well.

If Maureen Downey and the Atlanta Journal Constitution want to say that evidence shows “zero impact” from vouchers, then they have to explain away all of this evidence.  And if they don’t want to justify their claims in the pages of their paper, we can hold them accountable on the web.

(edited for typos)

1 Comment | vouchers | Tagged: , , , | Permalink
Posted by Jay P. Greene

Voucher Effects on Participants

August 21, 2008

(This is an update of a post I originally wrote on August 21.  I’ve included the new DC voucher findings.)

Here is what I believe is a complete (no cherry-picking) list of analyses taking advantage of random-assignment experiments of the effect of vouchers on participants.  As I’ve previously written, 9 of the 10 analyses show significant, positive effects for at least some subgroups of students.

All of them have been published in peer reviewed journals or were subject to outside peer review by the federal government.

Four of the 10 studies are independent replications of earlier analyses.  Cowen replicates Greene, 2001.  Rouse replicates Greene, Peterson, and Du.  Barnard, et al replicate Peterson and Howell.  And Krueger and Zhu also replicate Peterson and Howell.  All of these independent replications (except for Krueger and Zhu) confirm the basic findings of the original analyses by also finding positive effects.

Anyone interested in a more complete discussion of these 10 analyses and why it is important to focus on the random-assignment studies, should read Patrick Wolf’s article in the BYU Law Review that has been reproduced here.

I’m eager to hear how Leo Casey and Eduwonkette, who’ve accused me of cherry-picking the evidence, respond.

  • These 6 studies conclude that all groups of student participants experienced reading or math achievement gains and/or increased likelihood of graduating from high school as a result of vouchers:

Cowen, Joshua M.  2008. “School Choice as a Latent Variable: Estimating the ‘Complier Average Causal Effect’ of Vouchers in Charlotte.” Policy Studies Journal 36 (2).

Greene, Jay P. 2001. “Vouchers in Charlotte,” Education Matters 1 (2):55-60.

Greene, Jay P., Paul E. Peterson, and Jiangtao Du. 1999. “Effectiveness of School Choice: The Milwaukee Experiment.” Education and Urban Society, 31, January, pp. 190-213.

Howell, William G., Patrick J. Wolf, David E. Campbell, and Paul E. Peterson. 2002. “School Vouchers and Academic Performance:  Results from Three Randomized Field Trials.” Journal of Policy Analysis and Management, 21, April, pp. 191-217. (Washington, DC: Gains for all participants, almost all were African Americans)

Rouse, Cecilia E. 1998. “Private School Vouchers and Student Achievement: An Evaluation of the Milwaukee Parental Choice Program,” The Quarterly Journal of Economics, 113(2): 553-602.

Wolf, Patrick, Babette Gutmann, Michael Puma, Brian Kisida, Lou Rizzo, Nada Eissa, and Marsha Silverberg. March 2009.  Evaluation of the DC Opportunity Scholarship Program: Impacts After Three Years. U.S. Department of Education, Institute of Education Sciences. Washington, DC: U.S. Government Printing Office. (In the fourth year report the sample size shrunk so that the positive achievement effect barely missed meeting a strict threshold for statistical significance — p < .06 just missing the bar of p < .05.  But this new report was able for the first time to measure the effect of vouchers on the likelihood that students would graduate high school.  As it turns out, vouchers significantly boosted high school graduation rates.  As Paul Peterson points out, this suggests that vouchers boosted both achievement and graduation rates in the 4th year.  Read the 4th year evaluation here.)

  • These 3 studies conclude that at least one important sub-group of student participants experienced achievement gains from the voucher and no subgroup of students was harmed:

Barnard, John, Constantine E. Frangakis, Jennifer L. Hill, and Donald B. Rubin. 2003. “Principal Stratification Approach to Broken Randomized Experiments: A Case Study of School Choice Vouchers in New York City,” Journal of the American Statistical Association 98 (462):299–323. (Gains for African Americans)

Howell, William G., Patrick J. Wolf, David E. Campbell, and Paul E. Peterson. 2002. “School Vouchers and Academic Performance:  Results from Three Randomized Field Trials.” Journal of Policy Analysis and Management, 21, April, pp. 191-217. (Dayton, Ohio: Gains for African Americans)

Peterson, Paul E., and William G. Howell. 2004. “Efficiency, Bias, and Classification Schemes: A Response to Alan B. Krueger and Pei Zhu.” American Behavioral Scientist, 47(5): 699-717.  (New York City: Gains for African Americans)

This 1 study concludes that no sub-group of student participants experienced achievement gains from the voucher:

Krueger, Alan B., and Pei Zhu. 2004. “Another Look at the New York City School Voucher Experiment,” The American Behavioral Scientist 47 (5):658–698.

(Update: For a review of systemic effect research — how expanded competition affects achievement in traditional public schools — see here.)

35 Comments | vouchers | Tagged: , , , , , , , , , , , , , , | Permalink
Posted by Jay P. Greene

Next Entries »