(Guest post by Greg Forster)
I don’t want to jump to conclusions about yesterday’s DC school voucher study, since the study is only just out and we haven’t had time to digest it. But something really caught my attention when I first read the grad rate result that Matt highlighted yesterday:
The major finding of this report, and it is MAJOR, is that students who were randomly selected to receive vouchers had an 82% graduation rate. That’s 12 percentage points higher than the students who didn’t receive vouchers.
Hold on! I thought to myself. That implies the control group’s graduation rate was seventy percent!
Sure enough, there it is, front and center in the study:
The offer of an OSP scholarship raised students’ probability of completing high school by 12 percentage points overall (figure ES-3). The graduation rate based on parent-provided information was 82 percent for the treatment group compared to 70 percent for the control group.
Seventy percent? I thought to myself. That doesn’t sound like the DC school system I know.
Sure enough, Education Week pegs the DC grad rate at forty-nine percent. So what’s the deal with this crazy control group?
Yes, one factor is that the control group is made up of “choosers” – families that sought out school choice. They’re likely to be systematically different from non-choosers, which is the whole reason we do these random assignment shindigs. But come on – they’re not that different.
Then it occurred to me – the 49% DC “public school” grad rate is for district schools; it doesn’t account for charter schools.
A whopping 38% of DC public school students are in charter schools. Now, given that the control group for this voucher study is made up entirely of “choosers,” what percent of that control group do you think are in charter schools? A lot higher than 38% is my guess. (I can’t seem to find data for this anywhere in the report – they spend so much time talking about how some of the private schools converted into charters, you would think they’d have found a few lines to talk about how some of the control group were in charters!)
[Update: HT to Brian for finding the figure (see comments) – 35% are in charters and 12% are in private schools. When I placed a bet that more than 38% would be in charter schools, I forgot that choosers would also choose private schools even without the voucher – which greatly strengthens my argument since private schools likely have even bigger effects on grad rates than charter schools.]
I mean, if your argument to explain the 70% grad rate in the control group is that choosers are very different from non-choosers, then doesn’t that very difference imply we should expect huge numbers of choosers who lose the voucher lottery to fall back on charter schools?
I hope you see where I’m going with this.
It seems obvious that if school choice improve graduation rates – which it clearly does, not only in this study but in previous ones in Milwaukee – then a lot of that benefit is being masked in this study because the control group is also excercising a lot of school choice!
What’s the real grad-rate benefit from school choice? Not “12 percentage points,” but somewhere between 12 percentage points and 33 percentage points.