Eduwonkette and Eduwonk Aren’t Edumarried?

The New York Sun had a nice profile yesterday of Eduwonkette.  Well, it’s not exactly a profile because Eduwonkette writes anonymously.  In the article some folks complain that her anonymity is a problem: “A co-director of the Education Sector think tank, Andrew Rotherham, suggested on his blog Eduwonk that Eduwonkette might be unfairly pretending to be unbiased because she has ‘skin in the game… It’s this issue of you got all this information to readers, without a vital piece of information for them to put it in context.'”

I think Andy’s mistaken on this. (Did they have some kind of edu-break-up?)  The issue is not who Eduonkette is, but whether she is right or not.  Knowing who she is does not make her evidence or arguments any more or less compelling.  I wish we all spent a whole lot less time analyzing people’s motives and a whole lot more time on their evidence and arguments. 

The only major problem with anonymity is lack of responsibility for being wrong.  There is a reputational price for making bad arguments or getting the evidence wrong that Eduwonkette avoids paying professionally — although she does pay a reputational price to the name brand of Eduwonkette.

Speaking of being wrong, Eduwonkette knocks the study Marcus Winters, Julie Trivitt, and I released today through the Manhattan Institute.  She complains: “It may be an elegantly executed study, or it may be a terrible study. The trouble is that based on the embargoed version released to the press, on which many a news article will appear today, it’s impossible to tell. There is a technical appendix, but that wasn’t provided up front to the press with the glossy embargoed study. Though the embargo has been lifted now and the report is publicly available, the technical appendix is not.”

This isn’t correct.  Embargoed copies of the study were provided to reporters upon their request.  If they requested the technical report, they could get that.  Both were available well in advance to reporters so that they could take time to read it and circulate it to other experts before writing a story.  Both the study and the technical report were made publicly available today (although there seems to be a glitch with the link to the technical report that should be fixed within hours).  The technical report can be found here.

And while we are on the subject of Eduwonkette being wrong, her attacks on test-based promotion policies are overdone.  The Jacob and Lefgren paper does raise concerns, but there is more positive evidence from the experience in Florida.  As I wrote in a previous post: “In a study I did with Marcus Winters that was published in Education Finance and Policy, we found that retained students significantly outperformed their comparable peers over the next two years.  In another study we published in the Economics of Education Review, we found that schools were not effective at identifying which students should be exempted from this test-based promotion policy and appeared to discriminate in applying these exemptions.  That is, white students were more likely to be exempted by school officials in Florida from being retained, but those students suffered academically by being exempted.”

Our results may actually be consistent with what Jacob and Lefgren find.  We find academic benefits for students retained in third grade.  They find: “that grade retention leads to a modest increase in the probability of dropping out for older students, but has no significant effect on younger students.”  It could be that test-based promotion is more beneficial when done with younger students.  It could also be that the policy has positive effects on achievement with some cost to graduation. 

And particularly severe problems with the integrity of test results used for promotion decisions in Chicago may limit the ability to generalize from Chicago’s experience.  In Chicago it may have been easier to move retained students forward by cheating on the next test than actually teaching them the basic skills they need to succeed in the next grade.

Besides, I’m sure that Edwuonkette wouldn’t put too much stock in Jacob and Lefgren’s non-peer-reviewed paper released straight to the public.  I’m sure she would be consistent in her view that: “By the time the study’s main findings already have been widely disseminated, some sucker with expertise in regression discontinuity may find a mistake while combing through that appendix, one that could alter the results of the study. But the news cycle will have moved on by then. Good luck interesting a reporter in that story… So as much as I like to kvetch about peer review and the pain and suffering it inflicts, it makes educational research better. It catches many problems and errors before studies go prime time, even if it doesn’t always work perfectly.”  

Or do these standards only apply to studies whose findings she doesn’t like?   If Eduwonkette isn’t careful she might get a reputation.

About these ads

6 Responses to Eduwonkette and Eduwonk Aren’t Edumarried?

  1. Doug says:

    Mr. Greene – if you have such high confidence in your work, why not go through the traditional route of honest research, and complete a full peer review process before trumpeting your findings to the media? There is a huge difference between Jacob and Lefgren posting a working paper online, and your own media blast.

    Eduwonkette may have applied a double standard here, and she may “pay a reputational price” for her posts. But let’s face it: Eduwonkette is a blogger. You (apparently) deliver causal claims about education policy to deliberative bodies as important as the U.S. Supreme Court. Whose work should we be holding to the higher standard?

  2. matthewladner says:

    Doug-

    The article Jay refers to in the post above has obviously been through peer review, as have several of his other articles. I sent a non-education related political science article into a journal a few years ago. The process took two years, and wound up with an audience (optimistically) dozens. The glacial pace of academic journals simply doesn’t square with the pace of policy debate.

    Conflating “peer review” with “honest research” is a big stretch as well. I’ll take the hurly-burly of think tank competition to the anonymity of peer review as an accountability practice any day of the week. Think of it as anonymous peer review versus public whoever feels like trying to tear you to bits today accountability.

  3. In addition to seconding Matthew’s comments, I think I’ve dealth with some of these issues in a new post here: http://jaypgreene.com/2008/07/08/eduwonkette-apologizes/

    Also, I should emphasize that the issue is not whether research should eventually make its way to a peer-reviewed outlet, which I have regularly done, but whether research should be kept from the public for years while policy debates are underway. It is now common practice among researchers to release working papers and reports to the public before they are finally published in peer reviewed journals.

  4. Doug says:

    These are fair comments – the peer-review process does indeed take a long time.

    On the other hand, the claim that reputation and the “marketplace of ideas” provide an appopriate and sufficient check on accuracy and honest research is completely off base. The MO of think-tank research as I see it is as follows: with the help of generous financial backing and an impressive PR/marketing machine, a report is blasted to the press. The education press–always looking for a headline–dutifully reports the results of this report. (It always helps when the report is issued by an official-sounding organization with a history of doing policy work, such as the Manhattan Institute, Fordham Foundation, Education Sector, etc). The punchline (vouchers, accountability, or what have you, “works”) rings throughout the policy community, Capitol Hill, and beyond. If, 1-2 years later, other scholars find methodological flaws, mistakes, or outright fraud–none of which I am suggesting apply here–the only people left to care are other scholars. The policy world and press have long moved on. Unless the academic community can band together, with the help of equally impressive funding, and blast the press with a correction to the original study, the correction will never see the light of day. And even if they could, I doubt that the press would find a correction or replication study very newsworthy. As long as they don’t, there are no reputational penalties to poor work.

    Jay, you have long been applying market principles to the world of education. Do you honestly see a working “market” for honest research here?

  5. [...] Greene and Eduwonkette???an anonymous education blogger whom Greene thinks is married to Eduwonk but I suspect is the original Wonkette???s kindergarten teacher???are having a tiff about [...]

  6. matthewladner says:

    Doug-

    I read an interesting article about the Columbia Journalism school a couple of years ago. The narrative of the article went along the lines of a new Dean came in and shook up the graduate program by requiring students to take statistical methods classes. The idea was to equip graduates of the nation’s most prestigious college of journalism with the tools to discern good research from bad.

    For the Dean’s trouble, he faced a rebellion on the part of some of his faculty. I couldn’t begin to understand why then or now. The reactionaries came across as complete Know-nothings. I hope the Dean ultimately prevailed.

    The river flows both ways on what you are describing. I can pick up the paper in any given week and find assertions about K-12 education that not only couldn’t stand up to peer review, they couldn’t stand up to simple logic or basic fact checking.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,567 other followers