1. Certain subgroups haven’t achieved statistically significant gains YET. Remember- the gains are incremental over time. There is no reason to expect all the various subgroups to reach the very high bar of statistical significance at the same time.

2. Vouchers are not a magic bullet, nor do they make you better looking or grant superpowers. They do help kids to learn to read better, make more satisfied parents for a fraction of the per pupil cost in DCPS.

Does the program have some positive effects? Absolutely, and I agree that it is too early in the study to cancel it. But this is not the silver bullet as that voucher supports purport. Moreover, the cost analysis of these programs have been infantile – how much would it cost if the vast majority of students decided to follow vouchers? What about the cost of shutting down old schools or building new ones as students move from district to district on a whim?

2) With any positive result, you can always break it down and find some identifiable group of students where the benefits don’t achieve statistical certainty. If you won’t call a program successful if you can do that to the results, you’ll never find a successful program.

So basically, the kids who actually did receive a voucher and used it had to make gains large enough to drag these other kids as a group over the level of statistical significance.

I don’t think this is a fair conclusion – the report explicitly shows results from both groups separately in all instances except for the first page summary (as is pretty standard procedure in any scientific study with many subgroups.

As to your specific point, “there was a statistically significant impact on reading
achievement of 4.5 scale score points (effect size (ES) = .13)8 from the offer of a
scholarship and 5.3 scale score points (ES = .15) from the use of a scholarship” a minor difference of less than half a month. As for math “There was no statistically significant impact on math achievement, overall (ES = .03)
from the offer of a scholarship nor from the use of a scholarship“. It’s clear that results are very similar for both groups, and no one is “dragging” anyone up statistically.

In fact, the study actually should be comparing between those that got the scholarship and those that used it – if you were looking at the effectiveness of a scholarship given to students who scored highly on their SATs, you would naturally compare its effect to those high SAT scorers who didn’t take the scholarship. If you compare this way (pg. 30) the results or only marginal, and some fail multiple test correction.

The most important finding, however, is this: “There were no statistically significant reading (ES = .05) or math (ES = .01)
achievement impacts for the high-priority subgroup of students who had attended a
SINI public school“. With the most important and at-risk group showed no improvements from the voucher program, how can you honestly call the program a success (or better yet, “nothing short of phenomenal” as you do over at NRO)? To me, it seems that the program helps those gifted students who would excel anyways, and offers nothing for the students who actually go to bad schools. Moreover, it seems to me that math improvement would be a better measurement of teaching methods than reading (which essentially improves if you just give students better books) and in that aspect the program failed entirely.

It’s an empirical question and we are going to find out one way or another!

And for the record, I support the IIT model as scientifically superior, but Matt is right that this model sets the bar high, so when the bar gets set high and the program clears it anyway, it should get due credit.