As head of a department that has received and may wish to continue receiving federal research funds, it is completely contrary to my self-interest to say this: the federal government should not be in the business of conducting or funding education policy research. The federal government should facilitate research by greatly expanding the availability of individual student data sets stripped of identifying information. But the federal government is particularly badly positioned to conduct or fund analyses based on those data.
The reasons for keeping the federal government out of education policy research should be obvious to everyone not blinded by the desire to keep eating at the trough. First, the federal government develops and advocates for particular education policies, so it has a conflict of interest in evaluating those policies. Even when those evaluations are outsourced to supposedly independent evaluators, they are never truly independent. The evaluation business is a repeat-play game, so everyone understands that they cannot alienate powerful political forces too much without risking future evaluation dollars. The safe thing to conclude in those circumstances is that the evidence is unclear about the effectiveness of a policy but future research is needed, which, not surprisingly, is what many federally funded evaluations find.
Unfortunately, political influence in education policy research is often more direct and explicit than the implicit distortions of a repeat-play game. Every federally funded evaluation with which I am familiar has been subject to at least some, subtle political influence.
I can’t mention most without breaking confidences, but I can briefly describe my own experience with a What Works Clearinghouse (WWC) panel on which I served (which was managed by a different firm than the one that currently manages WWC). On that panel we were supposed to identify what was known from the research literature on how to turn around failing schools. As we quickly discovered, there was virtually nothing known from rigorous research on how to successfully turn around failing schools. I suggested that we should simply report that as our finding — nothing is known. But we were told that the Department of Education wouldn’t like that and we had to say something about how to turn around schools. I asked on what basis we would draw those conclusions and was told that we should rely on our “professional judgment” informed by personal experience and non-rigorous research. So, we dutifully produced a report that was much more of a political document than a scientific one. We didn’t know anything from science about how to turn around schools, but we crafted a political answer to satisfy political needs.
In addition to being politically influenced, federally funded research is almost always overly expensive. The cost of federal education policy research is many-fold more expensive than that research has to be. There are several federal evaluations where the cost of the evaluation rivals the annual cost of the program being evaluated.
Beyond being politically distorted and cost-inefficient, a whole lot of federally funded research is really awful. In particular, I am thinking of the work of the federally funded regional research labs. For every useful study or review they release, there must be hundreds of drek. The regional labs are so bad that the Department of Education has been trying to eliminate them from their budget for years. But members of Congress want the pork, so they keep the regional labs alive.
Being politically distorted, cost-inefficient, and often of low quality is not a good combination. Let’s get the feds out of the research business. They can still play a critically important role of providing data sets to the research community, but they should not be funding evaluations or research summaries. We need the feds to help with data because privacy laws are too great of a barrier for individual researchers. But once basic data is available, the cost of analyzing the data should be quite low — just the time of the researchers and some computer equipment, perhaps supplemented with additional field data collection. And if there is no “official” evaluation or “official” summary of the research literature, the research community is free to examine the evidence and draw its own conclusions. Yes, there will be disagreement and messiness, but the world is uncertain and messy. Freedom is uncertain and messy. The solution is not to privilege over-priced, often lousy, politically driven federally funded work.
(edited for typos)
Jay P. Greene's Blog 
Yowza! Preach it, brother! Preach it!
Some good critiques of the subtle political influences in federal research that I think ought to be taken seriously, but I think the ultimate conclusion here is a little extreme.
First, I think there are some babies swimming around in that bathwater. IES has pushed the RELs to conduct more rigorous research and many of the national evaluations conducted by IES itself (or its contractors) are really quite rigorous.
Second, let’s not be naive — political influence shows up in non-federally funded research too. Do you think that research funded by curriculum developers is preferable to federally funded research? And lets not forget that many “academics” moonlight as commercial developers (or have dreams of turning their pet theories into marketable products). The federally funded studies might have to be “diplomatic” in the language they use to report null or negative findings, but at least they do report the findings, AND the freedom of information act requires that the findings be reported — unlike studies conducted by academics or developers, federal studies cannot just be filed in a desk drawer.
The bottom line is that political influences are a part of the life of every researcher who isn’t a member of the landed aristocracy, because all researchers need funding from somebody. Federally-funded research may not be perfect, but neither are the realistic alternatives.
The problem is not necessarily that federally funded research is more biased, but that it is more privileged. People know that they have to take research conducted by curriculum developers with a large dose of salt, but they have no idea how much monkey business occurs with federally funded work. Let the market of research ideas operate.
And the federal government can have a file drawer bias. Remember how the Head Start evaluation was delayed for several years: https://jaypgreene.com/2010/01/13/head-start-basically/
Lastly, on the regional labs… I’ve been hearing that they are getting better for more than a decade. I can’t think of more than 3 rigorous studies done by any of the RELS (funded with tens of millions of dollars) in the last 5 years. Can you?
[…] a comment » Jay P. Greene thinks the government should get out of educational policy research: …[T]he federal […]
What would a “market of ideas” look like in education research in the absence of federal support? Who funds the supply of ideas? Curriculum developers, school districts, and teacher unions have little incentive to produce unbiased research — they all just want to hear that they’re doing a great job. Collectively, parents have an incentive to pay for it, but what incentive-compatible mechanism exists for them to pool their resources to pay for it? The free-rider problem makes that almost impossible. This strikes me as a classic case of a public good. As with other public goods, the federal provision of this public good is not perfect, but I think it is better than nothing.
Regarding head start — I don’t dispute that. And I’m aware of other examples. But the point is that you know about it and there is legal recourse to force findings to be released eventually. That doesn’t exist for other types of research. If I download a data set, run some regressions, and don’t like the answers I get, nobody need ever know that I even contemplated the analysis, let alone put the results in my desk drawer. With federal studies there is a public paper trail and the findings must eventually be released. Not perfect, but what’s the superior alternative?
Regarding the RELs — I know they don’t have a great history, but right now there are over 20 RCTs being conducted by the RELs. Here’s an example:
Click to access REL_20094068.pdf
This evaluation found no impact of the evaluated product, and is up front about that finding. Who else other than the federal government would have funded this and all the other RCTs that the RELs are doing and that NCEE has been conducting over the past decade?
I’m not wedded to federal research, and I don’t deny that there are weaknesses. I just don’t see the realistic alternative.
Government should supply the information, other entities can fund and conduct the research – its not like the private sector NEVER figured anything out on its own before.
As for FOIA, good luck. Here in Nevada we had to turn to the ACLU to fight for some public records because the government tried to charge a parent $5,000 to get hands on public records – $5,000 for their “effort” at recovering the information.
The next best method for the government to ignore FOIA is to get their lawyers to wiggle out of it – with no public interest law firm or DA enforcing information requests (ours doesn’t even enforce open meeting laws) it is pretty easy to get away with hiding public records.
The last line of defense is to ignore the request as long as possible.
Of course, you can always summon up $500,000 in taxpayer money during the middle of a recession in which you claim 2,000 teachers will be laid off unless taxes are raised, in order to fund a study on which taxes should be raised…
🙂
John — How do we know what to believe about anything without the federal government funding the answer? The market of ideas, with all of its imperfections, works reasonably well.
For example, how do we know what to believe about various religions? Shouldn’t we have a federally funded What Works in Religion Clearinghouse to protect us from all of the self-interested arguments made about religion?
[…] Feds And Research Shouldn’t Mix, writes researcher Jay P. Greene. […]
My response went over-long. In the policy environment Professor Greene advocates, I would not trust Fed raw data any more than I currently trust DOE analysis. 100% agreement about the Regional Education Labs.
To parents: trust your instincts and homeschool. You will not do worse than the current system.
Jay,
It seems to me that there is a distinction between private and public goods. I generally see little reason for the federal government to involve itself in the provision of private goods. There is a great deal of research that falls under the category of “private good”, and I do not think it would be appropriate for the government to fund those types of research. It’s not clear to me that education research is a private good. If you disagree, perhaps you could make an argument as to why.
John,
Let’s be clear that we are talking about education policy research and not education itself. It seems to me that education policy research is no different from tax policy research, health care policy research, energy policy research, etc… There are plenty of private resources available to produce research on all of these policy questions without the need for the federal government to directly fund or conduct it. And once most of the data is provided, the cost of conducting research should be quite low.
Yes, all research on policy has some qualities of public goods (no one can be excluded from consuming it and that consumption does not diminish its availability to others), but it has enough of the character of private goods that it need not be directly subsidized to exist. Besides, not all public goods require federal funding.
The “public goods” argument does not imply the conclusions which many people (even some academic economists) imagine it implies: that society at large benefits from tax subsidy for or State production of the good.
A “public good” is a good which is (a) non-rival in consumption and (b) non-exclusive in production.
Non-rival means that my consumption of the good does not reduce the amount available to you. Non-exclusive means that producers of the good in an uncoerced market cannot charge consumers. A lighthouse in the classic example. Broadcast radio qualifies. As the example of broadcast radio indicates, producers of a public good (e.g., broadcast music) may find ways to get people (advertisers) to pay for the production of the good. A pretty girl walking down the street, radiating beauty, is a public good (she brings a twinkle to the eye of this old man, anyway), but neither State cosmetics subsidies nor State operation of beauty parlors are necessary, since women have a strong incentive (dinner, movies, etc.) to provide this good.
Why suppose that schools, textbook publishers, and education software developers would have insufficient incentive to seek education research? They do not need to compete in the comprehension (possession) of the research, but in the implementation.
Handel and Bach composed in the era before copyright.
Further, some goods are “public goods” until the State produces them. Newspapers, electronic broadcast news, and, I would argue, History and Civics instruction become public bads (State-worshipful propaganda) when the State produces them.
Finally, corporate oversight is a public good (all shareholders benefit from research that keeps executives honest). The State itself is a corporation. Therefore, oversight of State functions is a public good which the State itself cannot provide. State assumption of responsibility for the prduction of public goods transforms the free rider problem at the root of public goods analysis but does not eliminate it.
Malcolm,
Your point is obvious, but nicely stated. I agree that being a public good is not a sufficient condition for a good to be provided by the government. The pretty girl walking down the street has enough of a private incentive to be pretty that we don’t really need to subsidize her.
And Jay, I agree that there are many examples of research that do not require federal funding. Indeed, there are many examples of research in education that do not require federal funding. Curriculum developers have a sufficient incentive to invest in R&D. Many large employers have an incentive to make investments in better understanding what skills their employees need and how best to develop those skills. And there are a myriad of decisions that local school district officials must make that are perhaps best informed by research they conduct themselves that applies specifically to their own context (that type of research comes close to the type of research that the RELs historically conduct — not very high quality, but of interest to local policymakers. That type of low quality research of high local interest probably should not be paid for by the federal government, and is an excellent argument, in my view, for doing away with the RELs if that were to be the RELs only focus).
But there are examples of education research that I think are best funded by the federal government. Questions of whether specific interventions actually work in a real world setting, for example, seem appropriate for the federal government to support. Curriculum developers are no more likely to share research findings with their customers that their product doesn’t work than tobacco companies were likely to share their research that nicotine is addictive. Prior to the commitment by IES to rigorous evaluations of these types of products, there was scant rigorous evidence (practically zero experimental evidence) as to the effectiveness of these types of products. There was a widespread belief among education researchers that experimental studies in education were prima facie impossible.
Now, I would never argue that it is physically impossible for the type of rigorous research into the real-world effectiveness of specific educational interventions to be funded by anyone other than the federal government. Some foundations do support this type of research, for example (although foundations are a little tricky — their tax status gives them a little bit of a hint of federal support). But the real-world experience has been that really rigorous evaluations of specific educational interventions did not happen very much prior to IES.
In light of this practical experience, I think that it is incumbent on anyone suggesting the total elimination of federal support of education research to suggest an alternative model for accomplishing the same ends. Ideally that alternative model would be well thought out and supported with practical experience.
Do you have a specific proposal for such a model?
Rigorous education research has been gradually developing over decades. Like all areas of science, the early work was crude, and then gradually people figured out better and better methods. This process was well underway long before the IES.
If the IES has made a special contribution that greatly accelerated the ongoing process of improving methods, it was data collection. And Jay is explicitly saying that function should continue. The IES’s contribution to analytical rigor, as distinct from rigor in the collection of data, has been minor.
John,
I’m still in favor of a large federal role in collecting and making data available to researchers, which is most of the cost of conducting research. I see no need for the federal government to directly fund the analyses of those data. The feds and states may indirectly subsidize that activity by generally supporting universities, where faculty have their own incentives to analyze education policies and disseminate results. I guess academics are like the pretty girl walking down the street, except they tend not to be so pretty.
I guess academics are like the pretty girl walking down the street, except they tend not to be so pretty.
GET THAT IMAGE OUT OF MY BRAIN!
Like Homsar as the modestly hot girl