I read an interesting article about the Columbia Journalism school a couple of years ago. The narrative of the article went along the lines of a new Dean came in and shook up the graduate program by requiring students to take statistical methods classes. The idea was to equip graduates of the nation’s most prestigious college of journalism with the tools to discern good research from bad.

For the Dean’s trouble, he faced a rebellion on the part of some of his faculty. I couldn’t begin to understand why then or now. The reactionaries came across as complete Know-nothings. I hope the Dean ultimately prevailed.

The river flows both ways on what you are describing. I can pick up the paper in any given week and find assertions about K-12 education that not only couldn’t stand up to peer review, they couldn’t stand up to simple logic or basic fact checking.

On the other hand, the claim that reputation and the “marketplace of ideas” provide an appopriate and sufficient check on accuracy and honest research is completely off base. The MO of think-tank research as I see it is as follows: with the help of generous financial backing and an impressive PR/marketing machine, a report is blasted to the press. The education press–always looking for a headline–dutifully reports the results of this report. (It always helps when the report is issued by an official-sounding organization with a history of doing policy work, such as the Manhattan Institute, Fordham Foundation, Education Sector, etc). The punchline (vouchers, accountability, or what have you, “works”) rings throughout the policy community, Capitol Hill, and beyond. If, 1-2 years later, other scholars find methodological flaws, mistakes, or outright fraud–none of which I am suggesting apply here–the only people left to care are other scholars. The policy world and press have long moved on. Unless the academic community can band together, with the help of equally impressive funding, and blast the press with a correction to the original study, the correction will never see the light of day. And even if they could, I doubt that the press would find a correction or replication study very newsworthy. As long as they don’t, there are no reputational penalties to poor work.

Jay, you have long been applying market principles to the world of education. Do you honestly see a working “market” for honest research here?

Also, I should emphasize that the issue is not whether research should eventually make its way to a peer-reviewed outlet, which I have regularly done, but whether research should be kept from the public for years while policy debates are underway. It is now common practice among researchers to release working papers and reports to the public before they are finally published in peer reviewed journals.

The article Jay refers to in the post above has obviously been through peer review, as have several of his other articles. I sent a non-education related political science article into a journal a few years ago. The process took two years, and wound up with an audience (optimistically) dozens. The glacial pace of academic journals simply doesn’t square with the pace of policy debate.

Conflating “peer review” with “honest research” is a big stretch as well. I’ll take the hurly-burly of think tank competition to the anonymity of peer review as an accountability practice any day of the week. Think of it as anonymous peer review versus public whoever feels like trying to tear you to bits today accountability.

Eduwonkette may have applied a double standard here, and she may “pay a reputational price” for her posts. But let’s face it: Eduwonkette is a blogger. You (apparently) deliver causal claims about education policy to deliberative bodies as important as the U.S. Supreme Court. Whose work should we be holding to the higher standard?