Jay P. Greene's Blog

As head of a department that has received and may wish to continue receiving federal research funds, it is completely contrary to my self-interest to say this:  the federal government should not be in the business of conducting or funding education policy research.  The federal government should facilitate research by greatly expanding the availability of individual student data sets stripped of identifying information.  But the federal government is particularly badly positioned to conduct or fund analyses based on those data.

The reasons for keeping the federal government out of education policy research should be obvious to everyone not blinded by the desire to keep eating at the trough.  First, the federal government develops and advocates for particular education policies, so it has a conflict of interest in evaluating those policies.  Even when those evaluations are outsourced to supposedly independent evaluators, they are never truly independent.  The evaluation business is a repeat-play game, so everyone understands that they cannot alienate powerful political forces too much without risking future evaluation dollars.  The safe thing to conclude in those circumstances is that the evidence is unclear about the effectiveness of a policy but future research is needed, which, not surprisingly, is what many federally funded evaluations find.

Unfortunately, political influence in education policy research is often more direct and explicit than the implicit distortions of a repeat-play game.  Every federally funded evaluation with which I am familiar has been subject to at least some, subtle political influence. 

I can’t mention most without breaking confidences, but I can briefly describe my own experience with a What Works Clearinghouse (WWC) panel on which I served (which was managed by a different firm than the one that currently manages WWC).  On that panel we were supposed to identify what was known from the research literature on how to turn around failing schools.  As we quickly discovered, there was virtually nothing known from rigorous research on how to successfully turn around failing schools.  I suggested that we should simply report that as our finding — nothing is known.  But we were told that the Department of Education wouldn’t like that  and we had to say something about how to turn around schools.  I asked on what basis we would draw those conclusions and was told that we should rely on our “professional judgment” informed by personal experience and non-rigorous research.  So, we dutifully produced a report that was much more of a political document than a scientific one.  We didn’t know anything from science about how to turn around schools, but we crafted a political answer to satisfy political needs.

In addition to being politically influenced, federally funded research is almost always overly expensive.  The cost of federal education policy research is many-fold more expensive than that research has to be.  There are several federal evaluations where the cost of the evaluation rivals the annual cost of the program being evaluated.

Beyond being politically distorted and cost-inefficient, a whole lot of federally funded research is really awful.  In particular, I am thinking of the work of the federally funded regional research labs.  For every useful study or review they release, there must be hundreds of drek.  The regional labs are so bad that the Department of Education has been trying to eliminate them from their budget for years.  But members of Congress want the pork, so they keep the regional labs alive.

Being politically distorted, cost-inefficient, and often of low quality is not a good combination.  Let’s get the feds out of the research business.  They can still play a critically important role of providing data sets to the research community, but they should not be funding evaluations or research summaries.  We need the feds to help with data because privacy laws are too great of a barrier for individual researchers.  But once basic data is available, the cost of analyzing the data should be quite low — just the time of the researchers and some computer equipment, perhaps supplemented with additional field data collection.  And if there is no “official” evaluation or “official” summary of the research literature, the research community is free to examine the evidence and draw its own conclusions.  Yes, there will be disagreement and messiness, but the world is uncertain and messy.  Freedom is uncertain and messy.  The solution is not to privilege over-priced, often lousy, politically driven federally funded work.

(edited for typos)