The U.S. Department of Education’s “What Works Clearinghouse” (WWC) is supposed to adjudicate the scientific validity of competing education research claims so that policymakers, reporters, practitioners, and others don’t have to strain their brains to do it themselves. It would be much smarter for folks to exert the mental energy themselves rather than trust a government-operated truth committee to sort things out for them.
WWC makes mistakes, is subject to political manipulation, and applies arbitrary standards. In short, what WWC says is not The Truth. WWC is not necessarily less reliable than any other source that claims to adjudicate The Truth for you. Everyone may make mistakes, distort results, and apply arbitrary standards. The problem is that WWC has the official endorsement of the U.S. Department of Education, so many people fail to take their findings with the same grains of salt that they would to the findings of any other self-appointed truth committee. And with the possibility that government money may be conditioned on WWC endorsement, WWC’s shortcomings are potentially more dangerous.
I could provide numerous examples of WWC’s mistakes, political manipulation, and arbitrariness, but for the brevity of a blog post let me illustrate my point with just a few.
First, WWC was sloppy and lazy in its recent finding that the Milwaukee voucher evaluation, led by my colleagues Pat Wolf and John Witte, failed to meet “WWC evidence standards” because “the authors do not provide evidence that the subsamples of voucher recipients and public school comparison students analyzed in this study were initially equivalent in math and reading achievement.” WWC justifies their conclusion with a helpful footnote that explains: “At the time of publication, the WWC had contacted the corresponding author for additional information regarding the equivalence of the analysis samples at baseline and no response had been received.”
But if WWC had actually bothered to read the Milwaukee reports they would have found the evidence of equivalence they were looking for. The Milwaukee voucher evaluation that Pat and John are leading has a matched-sample research design. In fact, the research team produced an entire report whose purpose was to demonstrate that the matching had worked and produced comparable samples. In addition, in the 3rd Year report the researchers devoted an entire section (see appendix B) to documenting the continuing equivalence of the matched samples despite some attrition of students over time.
Rather than reading the reports and examining the evidence on the comparability of the matched samples, WWC decided that the best way to determine whether the research met their standards for sample equivalence was to email John Witte and ask him. I guess it’s all that hard work that justifies the multi-million dollar contract Mathematica receives from the U.S. Department of Education to run WWC.
As it turns out, Witte was traveling when WWC sent him the email. When he returned he deleted their request along with a bunch of other emails without examining it closely. But WWC took Witte’s non-response as confirmation that there was no evidence demonstrating the equivalence of the matched samples. WWC couldn’t be bothered to contact any of the several co-authors. They just went for their negative conclusion without further reading, thought, or effort.
I can’t prove it (and I’m sure my thought-process would not meet WWC standards), but I’ll bet that if the subject of the study was not vouchers, WWC would have been sure to read the reports closely and make extra efforts to contact co-authors before dismissing the research as failing to meet their standards. But voucher researchers have grown accustomed to double-standards when others assess their research. It’s just amazingly ironic to see the federally-sponsored entity charged with maintaining consistent and high standards fall so easily into their own double-standard.
Another example — I served on a WWC panel regarding school turnarounds a few years ago. We were charged with assessing the research on how to successfully turnaround a failing school. We quickly discovered that there was no research that met WWC’s standards on that question. I suggested that we simply report that there is no rigorous evidence on this topic. The staff rejected that suggestion, emphasizing that the Department of Education needed to have some evidence on effective turnaround strategies.
I have no idea why the political needs of the Department should have affected the truth committee in assessing the research, but it did. We were told to look at non-rigorous research, including case-studies, anecdotes, and our own experience to do our best in identifying promising strategies. It was strange — there were very tight criteria for what met WWC standards, but there were effectively no standards when it came to less rigorous research. We just had to use our professional judgment.
We ended up endorsing some turnaround strategies (I can’t even remember what they were) but we did so based on virtually no evidence. And this was all fine as long as we said that the conclusions were not based on research that met WWC standards. I still don’t know what would have been wrong with simply saying that research doesn’t have much to tell us about effective turnaround strategies, but I guess that’s not the way truth committees work. Truth committees have to provide the truth even when it is false.
The heart of the problem is that science has never depended on government-run truth committees to make progress. It is simply not possible for the government to adjudicate the truth on disputed topics because the temptation to manipulate the answer or simply to make sloppy and lazy mistakes is all too great. This is not a problem that is particular to the Obama Administration or to Mathematica. My second example was from the Bush Administration when WWC was run by AIR.
The hard reality is that you can never fully rely on any authority to adjudicate the truth for you. Yes, conflicting claims can be confusing. Yes, it would be wonderfully convenient if someone just sorted it all out for us. But once we give someone else the power to decide the truth on our behalf, we are prey to whatever distortions or mistakes they may make. And since self-interest introduces distortions and the tendency to make mistakes, the government is a particularly untrustworthy entity to rely upon when it comes to government policy.
Science has always made progress by people sorting through the mess of competing, often technical, claims. When official truth committees have intervened, it has almost always hindered scientific progress. Remember that it was the official truth committee that determined that Galileo was wrong. Truth committees have taken positions on evolution, global warming, and a host of other controversial topics. It simply doesn’t help.
We have no alternative to sorting through the evidence and trying to figure these things out ourselves. We may rely upon the expertise of others in helping us sort out competing claims, but we should always do so with caution, since those experts may be mistaken or even deceptive. But when the government starts weighing in as an expert, it speaks with far too much authority and can be much more coercive. A What Works Clearinghouse simply doesn’t work.
Jay P. Greene's Blog 
Jay,
I can’t speak for the review of the Milwaukee vouchers or for WWC review of instructional materials, where I myself had sometimes nagging questions in the back of my mind. It would be a wonderful idea to have someone serious to regularly review WWC research and — I am sure of that — it will just improve WWC.
I disagree with the analysis of your own participation, though. That was done in a different context, when writing the so-called “practice guides” rather than research summaries and reviews. Their purpose is different (see http://ies.ed.gov/ncee/wwc/publications/practiceguides ) and they try to provide the best practical answers at the time to important questions that cannot wait forever for conclusive research. Consequently they rely on three levels of evidence — weak, medium, and strong — and on the opinions of experts in the field. The report you participated in (http://ies.ed.gov/ncee/wwc/pdf/practiceguides/Turnaround_pg_04181.pdf) recommended only 4 practices for turnaround: (a) Signal the need for dramatic change with strong leadership (b) Maintain a consistent focus on improving instruction, (c) Make visible improvements early in the school turnaround process, and (d) Build a committed staff. It tagged all these practices as having WEAK evidence, the lowest level based predominantly on expert opinion as described in the report.
I don’t find that report, its recommendations, or similar, either wrong or misleading. In fact, I find it useful. IES could not have been more clear in reporting the limited evidence for those four recommendations.
I agree with you that caution should be always exercised. I also can’t vouch for other incidents, but it would help your case if you were to provide more specific examples along the line of the Milwaukee vouchers where WWC sloppiness is clear.
Hi Ze’ev — I agree that the turnaround report came out as mostly harmless. But why should the U.S. Department of Ed be in the business of picking reform strategies, especially when it has no real evidence on which to base its endorsement? Can’t we just let state and local officials figure this out for themselves? It is not as if those 4 recommendations were so difficult for people to discover on their own.
But the more basic point is that the federal government should not go around telling people what is true about research — even if they mostly do no harm and apply plenty of caveats.
Hmmm… How do you feel about NIH issuing guidance? The FDA?
ED tries to provide assistance with questions that seem to be important; that have states asking for guidance about them. That is one of its roles, to provide such guidance. ED does not try to enforce it.
I think it is helpful when such guidance clearly and transparently summarizes the extant research on important topics, including clear statements about absence of strong research. There are enough educational quacks around peddling “research” on anything and everything, and most states actually have difficulties separating the wheat from the chaff.
But it cannot be done sloppily, as your Milwaukee story indicates.
At first I thought this blog post was insightful and compelling but then I realized that it hadn’t been vetted by the U.S. Department of Blog-Post Truthiness. How do I know whether or not to believe it?
Hi Ze’ev — I’d actually rather not have the FDA or NIH telling me what to do. But even if you aren’t willing to go that far, you might agree that there is a difference between research on medical issues with less policy implication than research on how to operate the government’s own $550 billion education industry.
We have perfectly fine private groups, like the Campbell Collaborative (http://www.campbellcollaboration.org/ ), who can help policymakers or practitioners if they are seeing expert assistance.
The problem with having an “official” arbiter of the truth is that it is certain to be more abusive with its influence.
[…] This post was mentioned on Twitter by Chan Stroman, Jay P. Greene. Jay P. Greene said: What Doesn't Work Clearinghouse: http://wp.me/peH0y-1Kh […]
I don’t want to drag this to death so I’ll just say that I agree with you, in that I’d rather not have NIH/FDA tell me what to do, but I wouldn’t object to them informing me about what they find. But IES is not telling us what to do — it just summarizes the state of education research. I think this has been the role of the office of education since its inception — gathering education statistics and dissemination of ed research.
The “telling” part is rather new, perhaps the last 30-40 years, and I am not its big fan either. But that’s not IES’ role.
I wish that IES were more like the Bureau of Labor Statistics. Just collect and disseminate the data but don’t conduct the research or adjudicate the truth. The BLS doesn’t tell us the “scientifically” supported tax policy and neither should the Dept of Ed.