Fun With Peer Review

December 9, 2014

PHD Comics

(Guest post by Greg Forster)

I may have to revise my opinion of Vox; they seem to have taken an interest in the weaknesses of the peer review system. Of course there are a lot of responsible peer-reviewed journals and, well, peers. But there a lot of the other kind as well, and we are long past the point where simply having gone through something called “peer review” ought to count for anything.

One story details how unscrupulous researchers can manipulate journals, including – amazingly – posing as their own reviewers. In highly specialized fields, journal editors may not know who the appropriate reviewers would be, so they rely – apparently uncritically in some cases – on the “recommended reviewers” supplied by the article authors. Who in some cases are simply the authors themselves using another email address. One scientist used 130 email accounts to create a vast, self-validating “peer review and citation ring”; 60 papers were recently retracted after a 14-month investigation uncovered the fraud. A total of at least 110 articles have been pulled in the last two years due to this type of fraud.

Get me off your email list

Figure 1 from the article “Get Me Off Your Fucking Mailing List”

Accepted for publication by the highly reputable International Journal of Advanced Computer Technology

But the other story is a lot better. It details how some journals now survive not by selling subscriptions or getting institutional support, but by charging a fee to publish your paper. They are apparently known as “predatory journals” because they spam the email universe looking for gullible (or, presumably, unscrupulous) people looking to break into publication. “Article mills” (after the analogous “diploma mills”) would seem a more appropriate name.

As you can see above, the “peer review” process becomes somewhat lax in these cases. One pair of scientists slapped the above-referenced article and began submitting it to peer review spammers. They were amused to discover that one journal accepted their article for publication. Another journal not only accepted but published an article (consisting of nonsense text) by Maggie Simpson and Edna Krabappel. It now sends the authors regular demands that they pay their $459 bill.

But it’s not just spam scammers – peer review controls are easy to get past even at some highly reputable publishers.

Are We Allowed to Be Neither Naive Nor Cynical?

July 31, 2013

(Guest post by Greg Forster)

I have a question. Am I permitted to be neither naïve nor cynical about the Tony Bennett emails? Or is there some sort of law that dictates I must be one or the other? Indiana StateImpact places me with the Bennett supporters while Matt seems to think I’m attacking Bennett (I’m not sure how else to interpret “haters gonna hate”). I don’t intend to be either.

I find it difficult to buy the new house line, and I will continue to find it difficult until someone asks Bennett the obvious question: “If this was a glitch in the system, as we are now being told, why did you seek to change the grade only for this one school?” Rick Hess didn’t ask him that question. Matt seems uninterested in asking it, and seems to think I’m a “hater” for asking it. Until that question is answered, I don’t see why I’m a “hater” for pointing out uncomfortable realities.

Is it really so scandalous, does it really make me a “hater,” to acknowledge the obvious fact that politicians are responsive to their donors? When government sets educational standards and has to do what Bennett himself calls a “face validity” test, it is going to know which schools are run by major donors and it is going to be sensitive to that fact. Good grief, are we this naïve?

What we have now is not “the rest of the story” but a failure to seek the rest of the story. Or am I somehow missing something?

On the other hand, Ze’ev and others seem to think I’m saying all standards are arbitrary and there’s no such thing as a rational public consensus. I’m not; I’m just trying to be realistic about what I called “the sausage-making nature of the process” when those standards are being cooked up behind closed doors by a government bureaucracy and its political allies, as opposed to standards that emerge organically from the give and take of a thriving marketplace of options. Technology standards emerge in the context of a system dominated by consumer choice. Educational standards should emerge in the same way.

My Own Personal Narcissus Index

April 19, 2013


(Guest post by Greg Forster)

Picking back up on our discussion of narcissism, I’m sure you’re all dying to know that my new Win-Win report was featured on John Stossel last night! While you other losers here on JPGB have been wasting your time on Twitter, I’m finally getting the undivided attention of millions that I’ve always known I deserved.

Oh, wait, sorry – I didn’t mean to bash Twitter, because . . . Stossel also tweeted my study. Twitter’s totally cool now!

0035 rotated square
In case you forgot what I look like.

Seriously, I’m always grateful when people bring attention to my work. Stossel highlighted the numbers for impact on public schools: 23 empirical studies have looked at how school choice impacts academic outcomes in public schools, of which 22 found a positive effect and one found no visible difference; no empirical study has ever found a negative impact. He also mentioned the numbers for racial segregation: eight studies, seven positive, one neutral; none negative. (Stossel’s description may have left viewers thinking those public school academic effect studies were participant effect studies – I know it’s hard to do justice to the details in the short time TV allows, but at least I can note the difference here.)

Hope others are finding the report useful – that unbroken line of zeros in the “negative effects” column can’t be publicized too widely!

Gerson Cites Voltron

April 4, 2013

voltron team (original)

(Guest post by Greg Forster)

As long as we’re on the subject of narcissism here on JPGB, may I note that Michael Gerson quoted our Voltron op-ed in Monday’s Washington Post?

But even small, restricted choice programs have shown promising results — not revolutionary but promising. Last year a group of nine leading educational researchers summarized the evidence this way: “Among voucher programs, random-assignment studies generally find modest improvements in reading or math scores, or both. Achievement gains are typically small in each year, but cumulative over time. Graduation rates have been studied less often, but the available evidence indicates a substantial positive impact. . . . Other research questions regarding voucher program participants have included student safety, parent satisfaction, racial integration, services for students with disabilities, and outcomes related to civic participation and values. Results from these studies are consistently positive.”

I’d tweet about it, but I’m too cool for Twitter.

A Good Media Week

March 29, 2013

This week the work of two of my colleagues in the Department of Education Reform was mentioned in national newspapers.  Patrick Wolf’s research finding that the Milwaukee voucher program increased high school graduation rates from 75% to 94% was mentioned in the Wall Street Journal.  And Bob Maranto’s work on the decline in New York City’s murder rate as a result of more effective policing was mentioned in David Brooks’ column in the New York Times.

Way to go!

Putting the Shoe on the Other Foot

June 18, 2012

I’m struck by how regularly I come across reporting in the media that contains obvious and unquestioned prejudice.  My mental test to detect this kind of prejudice is to switch the named group to see if we would find the same phrasing acceptable if it were applied to another group.  Since the truth of the claim is usually irrelevant to the prohibition of certain phrasings as offensive, the test is not whether the claim is true for another group but whether it would be unacceptable regardless of its truth.

I thought of this recently when the CBS Sunday Morning show had a segment on how boys were doing significantly worse in school.  Kenyon College’s Dean Jennifer Delahunty was asked to help explain this phenomenon and here is what she said: “There’s a kind of anti-intellectualism of young men that really bothers me, that it’s not cool to be smart. That it’s not cool to be engaged. That it’s not cool to do your homework. That bothers me.”

Sociologist Michael Kimmel offered this: “Boys think that academic disengagement is a sign of masculinity. The less you can do in school, the less connected you are, the less interested you are, the more manly you are.”

For all I know these are true explanations and boys really are suffering academically because of a cultural mindset that associates masculinity with anti-intellectualism and opposition to academic effort or engagement.

But let’s apply my little test to see if we might find this phrasing acceptable if it were applied to explaining why girls do worse on some academic outcome.  Let’s just switch the words so that the experts said: “There’s a kind of anti-intellectualism of young women that really bothers me that it’s not cool to be smart. That it’s not cool to be engaged. That it’s not cool to do your homework. That bothers me.” or “Girls think that academic disengagement is a sign of femininity. The less you can do in school, the less connected you are, the less interested you are, the more feminine you are.”

A CBS reporter would never quote experts saying this as a plausible explanation for why girls were doing worse academically.  That would have to be explained by discrimination — factors outside of the control of girls.  But for boys saying that the problem is their masculinity is perfectly fine.

Obviously, there are acceptable prejudices in our society.  The problem is not the existence of those prejudices, since some may in fact be supported by evidence, but that there is a wide-spread dogma about which prejudices are acceptable based on nothing having to do with evidence.  I guess I would say that there is a kind of anti-intellectualism among reporters that really bothers me, that it’s not cool to think critically about their prejudices.

Jay Mathews Comes Back for More

May 29, 2012

(Guest post by Greg Forster)

True story: At the house of some friends from church, the elder son (about six years old) was explaining the big bandage he was wearing. He told us he had climbed up on the stove in order to reach the cookies that were on top of the refrigerator, accidentally turned on the range with his foot, fell over, and was badly burned.

The following exchange occurred:

ME: Did you learn a lesson from what happened?

HIM: Uh . . . no.

Apparently Jay Mathews didn’t learn anything either after getting badly burned on the stove of my wrath last year.

He’s once again up to his typical stove-climbing antics, still trying to reach the cookies of bipartisan acceptability on top of the refrigerator of political ambiguity. Over the weekend, he wrote:

Instead, the two parties pound each other with an education issue that makes them look tough to their most partisan supporters. That convenient weapon is vouchers, tax-supported scholarships for students who want to attend private schools. Obama has cut funds for a voucher program in the District, so Romney embraces it. “It will be a model for parental choice programs across the nation,” he said in the speech.

The split doesn’t affect the bipartisan approach to schools much because vouchers have no chance of ever expanding very far. There aren’t nearly enough available spaces in good private schools to meet the demand. Any significant growth in vouchers would lead to heavy government interference in private schools and kill any allegiance conservative Republicans had to it.

Let’s take these claims one by one:

vouchers have no chance of ever expanding very far

Uh, yeah, let me just go ahead and link this again. Thanks. If Mathews wants to lose another bet on vouchers’ legislative prospects, he’s welcome to as much pain as he wants.

He links that statement to an older article of his on the DC voucher program, which serves under 2,000 kids. Compare that to the gargantuan sizes of the new Indiana and Louisiana programs (400,000 kids eligible in Louisiana!).

I’m not saying we’ve reached the promised land, but the political trend is very obviously up and not down.

There aren’t nearly enough available spaces in good private schools to meet the demand.

William F. Buckley once asked, speaking about a person whose name escapes me: “What do you think he would do if the devil removed the blinders from his eyes and showed him the world of economics? I say the devil, because God would never be so cruel.”

What do you think Jay Mathews would do if the devil removed the blinders from his eyes and showed him that quantity supplied can change in response to demand?

Any significant growth in vouchers would lead to heavy government interference in private schools and kill any allegiance conservative Republicans had to it.

Yeah, except for the part where there are now 34 school choice programs serving 212,000 students, and this story Mathews is telling hasn’t happened anywhere.

Keep reaching for those cookies, Jay. You’ll get them someday.

(Edit: In the first version of this post, the devil made me write the wrong name in the WFB quote above.)