Comments on: What Does the Red Pill Do If I Don’t Take It?

By: Greg Forster

Greg Forster — Thu, 26 Jun 2008 20:06:49 +0000

What do you mean, the Borg are no more? They’re still out there. The loss of a few stray cubes here and there is of little long-term consequence to them. And someday no doubt they’ll get us all.

But of course, in another sense, resistance isn’t futile even if it only delays the conquest.

By: Brian Kisida

Brian Kisida — Thu, 26 Jun 2008 19:31:16 +0000

I can’t add much to what Greg and Marcus have already rightly pointed out. The study is ultimately an evaluation of expanded choices, and everyone on the treatment group was given that. It is not an evaluation of private schooling. I will, however, question Greg’s reliance on the Borg for wisdom. Resistance is not futile. If I remember right, Piccard ultimately resisted, and the Borg are no more.
Furthermore, I can point out that the ITT analysis had the same significant subgroup findings (albeit with less magnitude) as the “hidden” IV analysis had. See pages 37-38 of the report.

By: Matthew Ladner

Matthew Ladner — Fri, 20 Jun 2008 11:38:09 +0000

Joanne has confirmed my suspcions: ITT could be the right way to answer certain questions, but it’s not the question that the vast majority of people want answered. Worse still, people think they are getting their question answered, when in fact, you have to dig into appendix E to find it.

By: Greg Forster

Greg Forster — Thu, 19 Jun 2008 21:00:44 +0000

Let me speak for all nit-pickers: If you want to know the answer to the question, “If you let low-income parents pick private schools, will their kids perform better?” then the appropriate method is to leave the voucher decliners in the sample. The key word in that question is “let.”

I think you mean you want to know this: “If low-income parents actually pick private schools using a voucher, will their kids perform better?”

Meanwhile, thanks for linking to my post today on NCLB, and for paying me the ultimate compliment for an education researcher: mistaking me for Jay Greene!

By: Joanne Jacobs

Joanne Jacobs — Thu, 19 Jun 2008 20:35:11 +0000

Let me speak for all journalists: The reporter wants to know how the voucher-using students did compared to the students who weren’t offered a voucher. This is the easiest issue for people to understand: If you let low-income parents pick private schools, will their kids perform better? The reporter also wants to know what percentage of students didn’t use the voucher because that raises the issue of whether there are enough accessible schools that appeal to parents.

Most reporters are weak on math skills, but I don’t think you’ll find many education writers who don’t care about kids getting educated. Assume good faith and lousy math skills.

By: Greg Forster

Greg Forster — Thu, 19 Jun 2008 18:29:28 +0000

Welcome to the blogosphere, Marcus! As Obi-Wan said, you’ve taken your first step into a larger world. And as the Borg said, resistance is futile.

By: Marcus Winters

Marcus Winters — Thu, 19 Jun 2008 17:30:36 +0000

Greg is right. Matt is wrong. (Sorry, Matt)

ITT is an evaluation of a voucher program, IOT is an evaluation of private schooling. IOT is only appropriate for this if you are going to have a voucher program that mandated private schooling — but that’s not what we have or would ever argue for. ITT is an evalution of the impact of the opportunity of private schooling, which is what a voucher is. In fact, to have an absolutely true random assignment study (obviously impossible) rather than having a lottery of those who apply you would just have a lottery of everyone in a school system and let them decide to use it or not. This would be a more accurate representation of a voucher system that operates in the context of a public school environment.

My main issue is using a Bloom adjustment as the IOT rather than an IV procedure. For some reason IES seems to hate IV, but I still have no idea why. Seems to me that Bloom throws useful information away. But that’s IOT, so of secondary importance anyway.

By: Greg Forster

Greg Forster — Thu, 19 Jun 2008 15:45:39 +0000

Right, because reporters would never buy the storyline that the Bush administration manipulates research for political ends.

The reporters would simply take dictation from their IES sources, report those claims as fact, and then for the sake of fair play they’d include a quote from our side where we say “But . . . see . . . if you . . . “

By: Matthew Ladner

Matthew Ladner — Thu, 19 Jun 2008 14:47:28 +0000

I’d love to see some IES person trying to convince a reporter “but….see…..if you include the kids who didn’t use the voucher, then the overall number goes below the standard level of significance!”

The reporters reaction is likely to be “what in the world are you talking about?” IES guy could try to wax poetic about the theory of research design, but any reporter worth their salt would bring this back to the central question “Do kids using vouchers learn more, or not?”

By: Greg Forster

Greg Forster — Thu, 19 Jun 2008 14:14:25 +0000

Well, those characerizations of the finding would have been wrong even if we set aside the intention to treat issue. The analysis doesn’t find that the vouchers didn’t work, it finds that the voucher students had higher test scores but we can only be 91 percent confident that this was due to vouchers and not a statistical fluke.

I’m not arguing that it doesn’t matter what you say because the media will only misrepresent it anyway. It’s true that no matter what you say the media will only misrepresent it. But it still matters what you say.

However, if being sensitive to media manipulation is the issue, consider what the media reaction would have been if the user analysis were highlighted and the intention to treat analysis relegated to a footnote. In that case, you’d have people in the IES talking on background to reporters about how the study was manipulated to produce the “right” result, because the appropriate analysis was buried in a footnote. Would that look any better?

I agree that both analyses provide useful information. The question here is which one is most relevant. And that depends on the purpose of the study; different things will be more relevant depending on your purpose.

In this case, the study was commissioned by Congress to evaluate the effectiveness of the program. Not the effectiveness of private schools for those who attend them, not the effectiveness of vouchers for those who use them, but the effectiveness of the voucher program simply as such. Lack of takeup is one thing that affects the effectiveness of the program.

As a digression, the empirical evidence supporting vouchers is so overwhelming that I’m not really worried that this study just barely failed to reach an arbitrary benchmark of statistical certainty. Anyone who wants a rundown of all the randomized studies on vouchers can find it here:

http://www.friedmanfoundation.org/friedman/newsroom/ShowNewsReleaseItem.do?id=20107