I thought that the exchange with Eduwonkette over the appropriateness of releasing research without peer review had run its course with my last post. But it seems that it will never end. Here is her latest post and here is the reply that I posted in her comment section:
Eduwonkette is attempting to change the subject. I’ve never disputed that peer review can help provide additional assurances to readers about quality. The issue is whether research ought to be available to the public even if it has not been peer reviewed. In attacking the release of my most recent study Eduwonkette seems to be arguing that it is inappropriate to release research without peer review, at least under certain conditions that she only applies to research whose findings she does not like. If she were going to be consistent, she would have to criticize anyone who releases working papers of their research, which would be almost everyone doing serious research.
What’s more, she is still trapped in a contradiction: she can’t say that we should analyze the motives of people who release research directly to the public when assessing whether it is appropriate, while she prevents analysis of her own motives because she blogs anonymously. As I have now said several times, either she drops the suggestion that we analyze motives or she drops her role as an anonymous blogger. If she refuses to resolve this contradiction, Ed Week should stop lending her their reputation by hosting her blog. Let her be inconsistent in blogging at the expense of her own anonymous persona and not drain the respectability of Ed Week.
Lastly, the comparison of the market for education policy information and the market for cars comes from my most recent post in our exchange, but she oddly does not credit me here. (See https://jaypgreene.com/2008/07/12/see-were-in-italy/ ) Her position seems to be that we ought to forbid (or at least shun) the sale of used cars without warranties (translation: research without peer review). My argument is that used cars without warranties come at a risk but there are compensating benefits. Similarly, non-peer-reviewed research has its risks but also its benefits.
UPDATE — My exchange with Eduwonkette continues although it seems increasingly pointless. Here is my (slightly edited) last comment on her site:
“Let’s make this very concrete. Was it inappropriate for Marcus Winters and I to release our social promotion findings in 2004 without peer review, or should we have waited until it had been peer-reviewed and published (in various forms) in 2006, 2007, and again in 2008? If the appropriate thing is to wait, would interest groups, editorial boards, and bloggers similarly hold their tongues until the additional evidence came in? Would policymakers hold off on decisions that might have come out differently if they had the suppressed information?
Would it have been OK to release in 2004 as long as we tried to make it obscure enough so that people were less likely to find it? What if interest groups, bloggers, etc… found our obscure finding and promoted them (as has happened with Jesse Rothstein’s paper)?
And in saying ‘working papers and thinktank reports are released for entirely different functions’ you are repeating your call for an analysis of motives. You’ve said that think tanks want to influence policy (bad motive) while academics are trying to advance knowledge with each other (good motive). But if academics are serving the public good, shouldn’t they ultimately want to influence policy? I am an academic who also releases working papers through a think tank. Does that make my motives good or bad? I think all of this analysis of motives is silly when the real issue is the truth of claims, not why people are making those claims. Calling for an analysis of motives is especially silly for someone who is trying to influence people anonymously. The fact that you are trying to influence people through a blog does not give you a free pass from having to be consistent on this.”