Vouchers: Evidence and Ideology

(Guest post by Greg Forster)


Lately, Robert Enlow and I at the Friedman Foundation for Educational Choice have had to spend a lot of time responding to the erroneous claims Sol Stern has been making about school choice. I honestly hate to be going up against Sol Stern right at the moment when he’s doing important work in other areas. America owes Stern a debt for doing the basic journalistic work on Bill Ayers that most journalists covering the presidential race didn’t seem interested in doing.


But what can we do? We didn’t choose this fight. If Stern is going to make a bunch of false claims about school choice, it’s our responsibility to make sure people have access to the facts and the evidence that show he’s wrong.


That’s why Enlow and I have focused primarily on using data and evidence to demonstrate that Stern’s claims are directly contrary to the known facts. It’s been interesting to see how Stern and his defenders are responding.


I’ve been saddened at how little effort Stern and his many defenders are devoting to seriously addressing the evidence we present. For example, all the studies of the effects of vouchers on public schools that were conducted outside the city of Milwaukee have been completely ignored both by Stern and by every one of his defenders I’ve seen so far. Does evidence outside Milwaukee not count for some reason? Since most of the studies on this subject have been outside Milwaukee, this arbitrary focus on Milwaukee is hard to swallow.


And what about the studies in Milwaukee? All of them had positive findings: vouchers improve public schools. Unfortunately, Stern and his critics fail to engage with these studies seriously.


Stern had argued in his original article that school choice doesn’t improve public schools, on grounds that the aggregate performance of schools in Milwaukee is still bad. His critics pointed out that a large body of high quality empirical research found that vouchers have a positive effect on public schools, both in Milwaukee and elsewhere. If Milwaukee schools are still bad, that doesn’t prove vouchers aren’t helping; and since a large body of high quality empirical research says they do help, the obvious conclusion to reach – if we are going to be guided by the data – is that other factors are dragging down Milwaukee school performance at the same time vouchers are pulling it upward.


If an asthma patient starts using medicine, and at the same time takes up smoking, his overall health may not improve. But that doesn’t mean the medicine is no good. I also think that there may be a “neighborhood effect” in Milwaukee, since eligibility for the program isn’t spread evenly over the whole city.


There’s new research forthcoming in Milwaukee that I hope will shed more light on the particular reasons the city’s aggregate performance hasn’t improved while vouchers have exerted a positive influence on it. The important point is that all the science on this subject (with one exception, in D.C., which I’ve been careful to take note of when discussing the evidence) finds in favor of vouchers.


In Stern’s follow-up defense of his original article, his “response,” if you can call it that, is to repeat his original point – that the aggregate performance of schools in Milwaukee citywide are still generally bad.


He disguises his failure to respond to his critics’ argument by making a big deal out of dates. He says that all the studies in Milwaukee are at least six years old (which is actually not very old by the standards of education research), and then provides some more recent data on the citywide aggregate performance of Milwaukee schools. But this obviously has nothing to do with the question; Stern’s critics agree that the aggregate data show Milwaukee schools are still bad. The question is whether vouchers exert a positive or negative effect. Aggregate data are irrelevant; only causal studies can address the question.


Of course it’s easy to produce more up-to-date data if you’re not going to use scientific methods to distinguish the influence of different factors and ensure the accuracy of your analysis. If you don’t care about all that science stuff, there’s no need to wait for studies to be conducted; last year’s raw data will do fine.


Weak as this is, at least it talks about the evidence. The response to our use of facts and evidence has overwhelmingly been to accuse school choice supporters of ideological closed-mindedness. Although we are appealing to facts and evidence, we are accused of being unwilling to confront the facts and evidence – accused by people who themselves do not engage with the facts and evidence to which we appeal.


Stern, for example, complains at length that “school choice had become a secular faith, requiring enforced discipline” and “unity through an enforced code of silence.” Apparently when we demonstrate that his assertions are factually false, we are enforcing silence upon him. (We’ve been so successful in silencing Stern that he is now a darling of the New York Times. If he thinks this is silence, he should get his hearing checked.)


Similarly, when Stern’s claims received uncritical coverage from Daniel Casse in the Weekly Standard, Enlow and Neal McCluskey wrote in to correct the record. Casse responded by claiming, erroneously, that Stern had already addressed their arguments in his rebuttal.


Casse also repeated, in an abbreviated form, Stern’s non-response on the subject of the empirical studies in Milwaukee – and in so doing he changed it from a non-response to an error. He erroneously claims that Stern responded to our studies by citing the “most recent studies.” But Stern cites no studies; he just cites raw data. It’s not a study until you conduct a statistical analysis to distinguish the influence of particular factors (like vouchers) from the raw aggregate results – kind of like the analyses conducted in the studies that we cite and that Stern and Casse dismiss without serious discussion.


Casse then praised Stern’s article because “it dealt with the facts on the ground” and accused school choice supporters of “reciting the school choice catechism.”


Greg Anrig, in this Washington Monthly article, actually manages to broach the subject of the scientific quality of one of the Milwaukee studies. Unfortunately, he doesn’t cite any of the other research, in Milwaukee or elsewhere, examining the effect of vouchers on public schools. So if you read his article without knowing the facts, you’ll think that one Milwaukee study is the only study that ever found that vouchers improve public schools, when in fact there’s a large body of consistently positive research on the question.


Moreover, Anrig’s analysis of the one Milwaukee study he does cite is superficial. He points out that the results in that study may be attributable to the worst students leaving the public schools. Leave aside that this is unlikely to be the case, much less that it would account for the entire positive effect the study found. The more important point is that there have been numerous other studies of this question that use methods that allow researchers to examine whether this is driving the results. Guess what they find.


Though he ignores all but one of the studies cited by school choice supporters, shuffling all the rest offstage lest his audience become aware of the large body of research with positive findings on vouchers, Anrig cites other studies that he depicts as refuting the case for vouchers. Like Stern’s citation of the raw data in Milwaukee, these other studies in fact are methodologically unable to examine the only question that counts – what was the specific impact of vouchers, as distinct from the raw aggregate results? (I’m currently putting together a full-length response to Anrig’s article that will go over the specifics on these studies, but if you follow education research you already know about them – the notoriously tarnished HLM study of NAEP scores, the even more notoriously bogus WPRI fiasco, etc.)


But Anrig, like his predecessors, is primarily interested not in the quality of the evidence but in the motives of school choice supporters. He spends most of his time tracing the sinister influence of the Bradley Foundation and painting voucher supporters as right-wing ideologues.


And these are the more respectable versions of the argument. In the comment sections here on Jay P. Greene’s Blog, Pajamas Media, and Joanne Jacobs’s site, much the same argument is put in a cruder form: you can’t trust studies that find school choice works, because after all, they’re conducted by researchers who think that school choice works.


(Some of these commenters also seem to be confused about the provenance and data sources of these studies. I linked to copies of the studies stored in the Friedman Foundation’s research database, but that doesn’t make them Friedman Foundation studies. As I stated, they were conducted at Harvard, Princeton, etc. And at one point I linked to an ELS study I did last year that also contained an extensive review of the existing research on school choice, but that doesn’t mean all the previous studies on school choice were ELS studies.)


What is one to make of all this? The more facts and evidence we provide, the more we’re accused of ignoring the facts and evidence – by people who themselves fail to address the facts and evidence we provide.


I’m tempted to say that there’s a word for that sort of behavior. And there may be some merit in that explanation, though of course I have no way of knowing. But I also think there’s something else going on as well.


One prominent blogger put it succinctly to me over e-mail. The gist of his challenge was something like: “Why don’t you just admit that all this evidence and data is just for show, and you really support school choice for ideological reasons?”


I think this expresses an idea that many people have – that there is “evidence” over here and then there is “ideology” over there, and the two exist in hermetically sealed containers and can never have any contact with one another. (Perhaps this tendency is part of the long-term damage wrought by Max Weber’s misuse of the fact/value distinction, but that’s a question for another time.)


On this view, if you know that somebody has a strong ideology, you have him “pegged” and can dismiss any evidence he brings in support of his position as a mere epiphenomenon. The evidence is a distraction from your real task, which is to identify and reveal the pernicious influence of his ideology on his thinking. Hence the widespread assumption that when a school choice supporter brings facts and evidence, there is no need to trouble yourself addressing all that stuff. Why bother? The point is that he’s an ideologue; the facts are irrelevant.


But, as I explained to the blogger who issued that challenge, evidence and ideology are not hermetically sealed. Ideology includes policy preferences, but those policy preferences are always grounded in a set of expectations about the way the world works. In fact, I would say that an “ideology” is better defined as a set of expectations about how the world works than as a set of policy preferences. (That would help explain, for example, why we still speak of differences between “liberal” and “conservative” viewpoints even on issues like immigration where there are a lot of liberals and conservatives on both sides.) And our expectations about how the world works are subject to verification or falsification by evidence.


So, for example, I hold an ideology that says (broadly speaking) that freedom makes social institutions work better. That’s one of the more important reasons I support school choice – because I want schools (all schools, public and private) to get better, and I have an expectation that when educational freedom is increased, schools will improve. My ideology is subject to empirical verification. If school choice programs do in fact make public schools better – as the empirical studies consistently show they do – then that is evidence that supports my ideology.


Even the one study that has ever shown that vouchers didn’t improve public schools, the one in D.C., also confirms my ideology. The D.C. program gives cash bribes to the public school system to compensate for lost students, thus undermining the competitive incentives that would otherwise improve public schools – so the absence of a positive voucher impact is just what my ideology would predict.


Other evidence may also be relevant to the truth or falsehood of my ideology, of course. The point is that evidence is relevant, and truth or falsehood is the issue that matters.


Now, as I’ve already sort of obliquely indicated, my view that freedom makes things work better is not the only reason I support school choice. But it is one of the more important reasons. So, if you somehow proved to me that freedom doesn’t make social institutions work better, I wouldn’t immediately disavow school choice, since there are other reasons besides that to support it. However, I would have significantly less reason to support it than I did before.


If we really think that evidence has nothing to do with ideology, I don’t see how we avoid the conclusion that people’s beliefs have nothing to do with truth or falsehood – ultimately, that all human thought is irrational. Bottom line, you aren’t entitled to ignore your opponent’s evidence, or dismiss it as tainted because it is cited by your opponent.


UPDATE: See this list of complete lists of all the empirical research on vouchers.


Edited for typos

14 Responses to Vouchers: Evidence and Ideology

  1. […] In response to comments here and elsewhere on his school choice argument, Greg Forster returns to the fray on Jay Greene’s blog to argue about Vouchers: evidence and ideology. […]

  2. Job says:

    I oppose vouchers because I do not want Ted Kennedy regulating Southern Baptist schools, which is what would happen because for working class and middle income families, the only the schools receiving the voucher money would survive – because they would be cheaper, offer more services, and have “the stamp of approval” from the state – and the schools that refuse the state money will go out of business. Once the government has eliminated the competition, it will come in with their usual batch of regulations: gay rights, feminism, environmentalism, abortion, political correctness, etc. This is already happening in a lot of states where the government is funding preschools … the independent preschools, daycares, etc. that refuse to take the money are going out of business. With a voucher system, “private schools” will actually be government schools …

  3. Greg Forster says:

    That’s certainly a good question to raise. However, it is worth looking at whether these predictions of government control have actually come to pass.

    In the 20 years of the modern school choice movement, since the Milwaukee voucher program was established, government control of participating private schools hasn’t been a problem in any school choice program. There are now almost 190,000 students attending private schools using vouchers and tax-credit scholarship programs, and so far 1) efforts to impose regulations on participating private schools have not been very ambitious in terms of the regulations they try to impose, and 2) even those efforts have largely failed to pass – they get proposed but they don’t get passed in the legislature.

    In fact, over time school choice programs are imposing fewer restrictions on participating private schools, not more. That reflects the dramatic increase in the political success of school choice.

    For more discussion of this issue, see the FAQ section of the publication “ABCs of School Choice,” published by the Friedman Foundation, and our recent report “Grading School Choice.”

  4. Sam Greenway says:

    Any comment on what’s going on in Minneapolis? Some unexpected choices made by students who have school choice:
    1-Go to school with best athletic program in your sport to better chances of athletic scholarship
    2-Go to EASIEST school to graduate from, if you’re not that keen on going to college

    Bottom line, if local community cares about education, the school will reflect that–and vice versa.

    I’m slowly reconsidering my pro-school-choice stance…

  5. […] Forster blows up the claim that ideology – not evidence – is behind the school choice movement, noting that a whole host of […]

  6. […] on Jay Greene’s blog today, at least one ardent supporter of school choice — the Friedman Foundation’s Greg Forster […]

  7. George Mitchell says:

    Forster’s commentary reflects the fact that he has a much more solid understanding of the evidence than Stern. It would be interesting to see him or others tackle the alternative that Stern and some others have advanced as to how public schools supposedly can turn themselves around absent a competitive environment.

    Those who have been on the ground in Milwaukee for a couple decades definitely hoped that by now public schools there would have shown a more positive response. But as Forster notes (and Stern obfuscates) there has been some improvement and the research suggests it would not have occurred in the absence of choice. To many the logical policy response would be to expand options, increase the number of parents eligible for them, and try to close somewhat the large gap in fiscal support available to public schools vs. the alternatives that now exist. The fact that schools in the choice program have been able to grow and largely thrive at a fraction of the cost of public schools is striking.

    It would be helpful if Stern stopped the bobbing and weaving and put forth a specific plan as to what he believes the Wisconsin Legislature and the Milwaukee Public Schools should do if, as he asserts, choice is not “working” as well as some had predicted. Would he reduce options? Increase them? Increase funding for choice schools? Reduce it? Mandate curricular “reforms”? The devil’s in the details. Stern is getting lots of attention for his Nixon to China moment but as Forster notes much of the attention is coming from media outlets and others who take pleasure in saying, “Hey look, Sol Stern says vouchers don’t work.”

  8. Checker Finn has a post over at Flypaper on the Washington Monthly article. The title of his post is “Choice is Winning.”
    See http://www.edexcellence.net/flypaper/index.php/2008/05/choice-is-winning/

  9. Greg Forster says:

    To Sam Greenway’s post, five responses:

    1) Are you talking about charter schools? Because I have a hard time believing you’re talking about Minnesota’s tax deduction for private school tuition. Just checking for the sake of clarity – the issues are different with charter schools.

    2) Do you have any data that would allow us to compare the number of students who are getting a worse education (because of their own bad choices) to the number of children who are getting a better education (because they were trapped in failing schools until they were given a choice)? Because I haven’t seen any data in Minnesota, but all the hard data in other places indicate that school choice provides a better education. See the research review section in this study:


    3) You don’t seem to have a high opinion of people’s ability to make choices for their children. I’ve already indicated that the data don’t support that. But even if we had no data, I would say that I would trust families to make better decisions for their children than a monopolistic government bureaucracy. How many parents would really choose a worse education for their children on purpose, just so their children wouldn’t have to work as hard?

    4) You write “if the local community cares about education, the school will reflect that.” This assumes that there is no benefit from offering choices. We can just “make the school better” and then it won’t matter that you have no choice where you go. But different children have different educational needs that can be served better in different ways. No one school can be the right school for every child. And the evidence is pretty clear that competition for students creates positive incentives that improve all schools.

    5) For that matter, why should my child’s education be held hostage to my community’s priorities? Suppose I live in a community that doesn’t care about education – say, a neighborhood with a lot of childless households that don’t want to pay high property taxes so that other people’s kids can have a better school. Why should my child’s life be crippled because my neighbors don’t value education?

  10. I would also note that there is a new study on whether vouchers increase achievement in public schools — the evaluation that Marcus Winters and I just released of the McKay Scholarship Program in Florida (vouchers for special ed). The study can be viewed here:

    Anyone claiming that vouchers don’t affect public school performance would also have to address this new evidence.

  11. […] The first part of this Greg Forster blog post is just more of the back and forth over Sol Stern’s recent article on vouchers but the second part is an interesting discussion of how ideology and research intersect. It’s worth checking out. […]

  12. […] at Jay Greene’s blog, Greg Forster writes a long (but good) essay about the quality of school choice research. It seems the more evidence comes out, the weaker […]

  13. tauna rogers says:

    In response to these remarks by Jay P. Greene on the McKay VOUCHER program:

    “Anyone claiming that vouchers don’t affect public school performance would also have to address this new evidence.”

    Check this out: http://www.epicpolicy.org/newsletter/2008/05/mckay-voucher-report-relies-flawed-analysis

  14. […] thing is getting pretty tiresome. Greene’s colleague Greg Forster has dealt with the phenomenon before, as has Cato???s Andrew Coulson and former AEI president […]

Leave a Reply to “Ideologues” Strike Back—with Evidence! | Think Tank West Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: