Beneath the Surface – DC Vouchers and Charters

(Guest post by Greg Forster)

I don’t want to jump to conclusions about yesterday’s DC school voucher study, since the study is only just out and we haven’t had time to digest it. But something really caught my attention when I first read the grad rate result that Matt highlighted yesterday:

The major finding of this report, and it is MAJOR, is that students who were randomly selected to receive vouchers had an 82% graduation rate.  That’s 12 percentage points higher than the students who didn’t receive vouchers.

Hold on! I thought to myself. That implies the control group’s graduation rate was seventy percent!

Sure enough, there it is, front and center in the study:

The offer of an OSP scholarship raised students’ probability of completing high school by 12 percentage points overall (figure ES-3). The graduation rate based on parent-provided information was 82 percent for the treatment group compared to 70 percent for the control group.


Seventy percent? I thought to myself. That doesn’t sound like the DC school system I know.

Sure enough, Education Week pegs the DC grad rate at forty-nine percent. So what’s the deal with this crazy control group?

Yes, one factor is that the control group is made up of “choosers” – families that sought out school choice. They’re likely to be systematically different from non-choosers, which is the whole reason we do these random assignment shindigs. But come on – they’re not that different.

Then it occurred to me – the 49% DC “public school” grad rate is for district schools; it doesn’t account for charter schools.

A whopping 38% of DC public school students are in charter schools. Now, given that the control group for this voucher study is made up entirely of “choosers,” what percent of that control group do you think are in charter schools? A lot higher than 38% is my guess. (I can’t seem to find data for this anywhere in the report – they spend so much time talking about how some of the private schools converted into charters, you would think they’d have found a few lines to talk about how some of the control group were in charters!)

[Update: HT to Brian for finding the figure (see comments) – 35% are in charters and 12% are in private schools. When I placed a bet that more than 38% would be in charter schools, I forgot that choosers would also choose private schools even without the voucher – which greatly strengthens my argument since private schools likely have even bigger effects on grad rates than charter schools.]

I mean, if your argument to explain the 70% grad rate in the control group is that choosers are very different from non-choosers, then doesn’t that very difference imply we should expect huge numbers of choosers who lose the voucher lottery to fall back on charter schools?

I hope you see where I’m going with this.

It seems obvious that if school choice improve graduation rates – which it clearly does, not only in this study but in previous ones in Milwaukee – then a lot of that benefit is being masked in this study because the control group is also excercising a lot of school choice!

What’s the real grad-rate benefit from school choice? Not “12 percentage points,” but somewhere between 12 percentage points and 33 percentage points.


14 Responses to Beneath the Surface – DC Vouchers and Charters

  1. S. Occerslame says:

    The information you seek is on page 27 of the report. 35% of the control group kids went to charters, and 12% of them went to private schools. So, yes, nearly half went to schools of choice.

  2. Greg Forster says:

    Thanks! I’ve updated the post. I must have missed that going through it with the text search.

  3. Patrick says:

    So the finding “no conclusive evidence that the OSP affected student achievement” could be distorted by the fact that 47 percent of the control group exercised school choice anyway?

  4. […] slightly hidden, very-high graduation rate in D.C. uncovered. (Jay Greene) Filed under: Newsroom  Print Share Posted at 7:27 […]

  5. […] The choice benefit is larger, writes Greg Forster. The graduation rate is only 49 percent in district-run schools. How did the […]

  6. Johnny Utah says:

    Patrick, the “no conclusive evidence” line is based upon a level of statistical significance that must be met according to the department of ed. Most academics, however, would accept that the level of statistical significance produced by the achievement score comparisons in the final study was, in fact, conclusive. For example, when economists Lisa Barrow and Cecilia Rouse published a prominent review of voucher research in 2008, they chose to code the second year reading results from DC as significant (p=.09 in that case), even though the dept of ed said there was no effect. The p-value in the third year of the DC evaluation was .01, and in the final year it was .06.

    The dept of ed’s arbitrary cutoff is .05.

    • Patrick says:

      Maybe I’m wrong, but I always thought that a p value of .05 or less was required for social science research (and .02 for hard science).

      • Johnny Utah says:

        That’s not exactly right Patrick. There certainly isn’t a law that’s enforced by the social science police. P-values of .10 are considered statistically significant all the time. And, in some of the most stats heavy journals, no p-value “cutoffs” are set. Cutoffs are arbitrary, and some feel simply reporting the p-values (or just the standard errors) gives readers more information than simply showing stars or the absence of stars.

  7. […] Writing on Jay Greene’s blog, Greg Forster deconstructs the control group (since the graduation rate for D.C. Public Schools is actually 49 percent), and concludes the grad-rate benefit from the voucher program is “somewhere between 12 percentage points and 33 percentage points.” […]

  8. tim-10-ber says:

    Ok — reading between the lines (without reading te report) students whose parents have the ability to exercise choice or have the option to exercise choice do better in school and tend to graduate. I agree with that.

    My question is and I really need help…what becomes of the kids in the public schools that do not have access to choice, do not have parents/significant adult in their life that believe in education, etc? Do zoned schools become the place where these kids get lost for life?

    Choice is here to stay. What is being done to help the kids that deserve the ability to hope, dream and obtain more in their lives?


  9. Greg Forster says:

    What’s being done for them? School choice is being done for them!

    That is, a large body of high-quality empirical studies consistently finds that school choice improves not only outcomes for the students who use it, but also outcomes at nearby public schools. That’s because choice introduces healthy competitive incentives that cause public schools to improve. For more, start here.

  10. Maximus says:

    How much of the rate has nothing to do with the difference between public and charter schools, but can be attributed to the fact that the “choosers” have parents that are more involved in their childrens education than the non-choosers?

  11. Greg Forster says:

    We don’t know. As I said, that’s why we do random assignment studies. It’s because we can’t answer that question quantitatively. We don’t even know which direction the variable goes. Should we expect choosers to have higher outcomes because their parents are more involved? Or should we expect choosers to have lower outcomes because it’s the kids who are having trouble whose parents will be more likely to seek out alternatives?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s