Fun With Peer Review

December 9, 2014

PHD Comics

(Guest post by Greg Forster)

I may have to revise my opinion of Vox; they seem to have taken an interest in the weaknesses of the peer review system. Of course there are a lot of responsible peer-reviewed journals and, well, peers. But there a lot of the other kind as well, and we are long past the point where simply having gone through something called “peer review” ought to count for anything.

One story details how unscrupulous researchers can manipulate journals, including – amazingly – posing as their own reviewers. In highly specialized fields, journal editors may not know who the appropriate reviewers would be, so they rely – apparently uncritically in some cases – on the “recommended reviewers” supplied by the article authors. Who in some cases are simply the authors themselves using another email address. One scientist used 130 email accounts to create a vast, self-validating “peer review and citation ring”; 60 papers were recently retracted after a 14-month investigation uncovered the fraud. A total of at least 110 articles have been pulled in the last two years due to this type of fraud.

Get me off your email list

Figure 1 from the article “Get Me Off Your Fucking Mailing List”

Accepted for publication by the highly reputable International Journal of Advanced Computer Technology

But the other story is a lot better. It details how some journals now survive not by selling subscriptions or getting institutional support, but by charging a fee to publish your paper. They are apparently known as “predatory journals” because they spam the email universe looking for gullible (or, presumably, unscrupulous) people looking to break into publication. “Article mills” (after the analogous “diploma mills”) would seem a more appropriate name.

As you can see above, the “peer review” process becomes somewhat lax in these cases. One pair of scientists slapped the above-referenced article and began submitting it to peer review spammers. They were amused to discover that one journal accepted their article for publication. Another journal not only accepted but published an article (consisting of nonsense text) by Maggie Simpson and Edna Krabappel. It now sends the authors regular demands that they pay their $459 bill.

But it’s not just spam scammers – peer review controls are easy to get past even at some highly reputable publishers.


Report Card on American Education Released Today

October 29, 2014

(Guest Post by Matthew Ladner)

The 19th Edition of ALEC’s Report Card on American Education: Ranking State Performance, Progress and Reform coauthored by yours truly and Dave Myslinski hit the presses today. Lots of good stuff in this year’s model, including an update of state rankings, a review of the first decade of universal NAEP participation, and a chapter focused on comparing the results of large urban districts.

So going up to the 30,000 level and back down, international results show that the United States is world-class in spending per pupil, not so much in learning per pupil, and that our results for Black and Hispanic students are closer to those of Mexico than of South Korea, despite the fact that Mexico has a far larger poverty problem and spends a small fraction of American spending.

The United States is making progress, but only an average amount of progress so we aren’t going to be catching up  much at the current pace. When you break down American results by state, you find that some states are pushing the national average cart, while others are riding in the cart. Which ones? Glad you asked:

4 NAEP exams

 

So the states in blue have made statistically significant gains in all four regular NAEP tests (4th and 8th grade reading and math) between 2003 and 2013.  Of the 21 states pulling that feat off, 14 are located in either the West or the South. The Midwest excepting MN, Great Plains, Mid-Atlantic, New York and Texas didn’t carry their weight on improvement (to varying degrees in general math gains were easier to come by than reading, 4th grade improvement easier than 8th grade) during this period. Michigan was the only state to make no significant progress on any of the four regular NAEP exams, a trend I hope they will reverse soon. All other states made progress on one or more of the exams. Note also that this map only shows improvement, few if any of the darkened states have internationally competitive scores, and the few that do tend to hold the good end of the stick on various achievement gaps.

So on the one hand, American education outcomes have never been higher than the 2013 NAEP.  On the other hand, no one yet has any cause for celebration. When we have any states that approach a Asian/European level of bang for the buck in learning outcomes, we’ll let you know about it, but thus far, not so much.

In Chapter 4 of the Report Card we take a close look at the Trial Urban District Assessment (TUDA) NAEP and apply the same “general education low-income” student comparison that we use in the states to improve comparability. Low-income general ed kids were seven times more likely to reach the Proficient level of 4th grade reading in Miami (the top performing district) as in Detroit (the lowest performing). Mind you have only a little better shot at 1 in 3 of scoring Proficient in Miami, so there are many miles to go. Looking at both 4th and 8th grade reading, Miami, New York City, Hillsborough County FL (Tampa) and Boston cluster near the top of the ratings. The District of Columbia does not (yet) rate near the top of the ratings, but their progress over time on NAEP is nothing short of remarkable since the mid 1990s. A large percentage of District students attend charter schools these days, and those charter schools show not only higher scores but also faster improvement than district schools, which are themselves improving.

In any case, slide on down to the following link if you want to see how your state is doing.

Indiana State page

 

 

 

 


Why is the man with the goatee smiling?

October 16, 2014

It might have something to do with this new report from MDRC showing a 9.4% increase in graduation rates in NYC in the “small high-schools” initiative. Students attending small high schools attended college at an 8.4% higher rate as well.

So just to review, Gates FF had a winning strategy on their hands- it had a plausible theory but not much empirical support. Sadly they dropped this strategy before waiting for empirical evaluations, which continue to pile up and have strongly positive results. The siren call of central planning lured them into an endless quagmire that also lacks empirical support (see Hanushek and Loveless) and also lacks a plausible theory of change. Small schools now lacks neither of these things.

There’s one obvious solution to all of this- he’s tan, rested and ready and he’s bringing back socks and sandals! Or perhaps it would be more accurate to say that he is bringing in socks and sandals for the first time. Regardless- bring back Tom Vander Ark!

 

 


Worse Than You Think

August 14, 2014

GIFSec.com

(Guest post by Greg Forster)

The most recent OCPA Perspectives carries my article on how public schools in Oklahoma (like everywhere else!) are worse than people think. I discuss some of the reasons that might lie behind the persistent tendency of the public to think that schools in their own neighborhoods are fine, and it’s always somebody else’s school that has problems:

Local pride creates an irrational bias to think that “our” schools must be good. Even more, it’s because parents would feel guilty admitting they’re sending their kids to schools that aren’t as good as they ought to be…This problem is itself one reason it’s horribly perverse to give government a school monopoly. It makes us feel like we can’t admit the system is failing without being failures ourselves.

These problems may be exacerbated in Oklahoma because it’s a relatively white and rural state. Our cultural image of “failing schools” seems to have been set in stone way back in the 1980s with movies like “Lean on Me.” It’s only the poor, black, inner-city schools that fail. White schools don’t fail. Suburban and rural schools don’t fail…Education reformers, unfortunately, have spent decades reinforcing this prejudice. We’ve typically used only one measurement of what counts as success in education reform: reducing the “achievement gap” between white, suburban schools and minority, urban schools. The unconscious assumption is that if a school is white and suburban, it must be succeeding. That kind of school must represent the best that American education is capable of. But why? Because it’s white and suburban?

I cite data from the new Harvard study “Not Just the Problems of Other People’s Children” and also data from the Global Report Card, put together by some guy I know. Check it out.


Fix Voucher Regulations with This One Weird Trick!

May 30, 2014

Public Rules on Private Schools

(Guest post by Greg Forster)

One of the big controversies surrounding school choice programs is whether they tend to increase government regulation of private schools. Big, sweeping claims have been easy to come by; serious scholarship studying the question, while not nonexistent, has been rare. Today the Friedman Foundation makes a major new contribution by releasing the study “Public Rules on Private Schools.” It is one of the most careful, methodical analyses to date on this question.

The big revelation for me in this study is that government regulations associated with voucher programs (as distinct from other types of school choice programs) is disproportionately made up of paperwork and other compliance requirements. Programs can largely nullify the effects of these regulations by adding some additional funding to cover compliance costs. Some programs do this already. This seems like a no-brainer for legislators to start including in future bill design.

So for the most part the war between voucher and tax-credit scholarship programs seems to me to be blown way out of proportion. Top up the voucher for compliance costs and the differences become unimportant.

Check out this awesome slideshow for tons of information plus author Drew Catt’s spot-on demonstration of what “nerd hipster irony” looks like.


Taste the ABC Rainbow!

January 23, 2014

2014 ABCs BLUE2014 ABCs GREEN2014 ABCs YELLOW2014 ABCs ORANGE2014 ABCs PINK

(Guest post by Greg Forster)

The new edition of the ABCs of School Choice is out – now available in a rainbow of colors, showing that Friedman provides the full spectrum of data on school choice programs.

No red, though? I’m disappointed.


It’s a Blowout: Tom Vander Ark 4, New Gates PLDD Strategy 0

August 29, 2013

The research evidence continues to pile up that the old Gates strategy of promoting small schools of choice has proven effective according to rigorous random-assignment design studies, while the new Gates PLDD strategy of building a national system of standards, assessments, and consequences has virtually no rigorous evidence to support it.

Under Tom Vander Ark’s leadership the Gates Foundation not only pursued an agenda based on a plausible theory of school improvement, but also initiated a series of high-quality studies to assess the results.  Even though Gates has largely abandoned its old strategy, those results are now pouring in.  We previously saw positive outcomes from a study by Lisa Barrow, Amy Claessens, and Diane Whitmore Schanzenbach of small schools in Chicago.  A non-Gates-funded analysis by my students, Anna Egalite and Brian Kisida, showed the same advantage for smaller schools in a national sample.  And in New York City, MDRC also demonstrated significant gains from small schools of choice.

Now MDRC has an updated analysis confirming that the benefits discovered earlier were extended and endured.  Randomly assigning students to small high schools “continue[s] to produce sustained positive effects, raising graduation rates by 9.5 percentage points.”  In addition, “more students are graduating ready for college: the [small high] schools raise by 6.8 percentage points the proportion of students scoring 75 or more on the English Regents exam, a critical measure of college readiness used by the City University of New York.”

What evidence do we have to support the new Gates PLDD strategy?  Umm, well, I’m sure Fordham can gather some of their friends together and give higher letter grades to states implementing the Gates PLDD strategy.  Over $6 million can buy some really good grades.

This is what the Gates Foundation has been reduced to — an organization that used to commission the most rigorous evaluations of their reform theory now invests overwhelmingly in the marketing and spinning of their new reform theory.  And they couldn’t even stick with the old reform theory of promoting small schools of choice long enough to see whether the rigorous evidence supported it.

Small schools?!?  That is like so 2007.  I think I’ll tweet my friends all about it, since Gates is now more interested in Twitter counts than random-assignment research.  Can Gates please put a grown-up in charge, like bringing back Tom Vander Ark?

[UPDATE — An alert reader notes that the score is actually 5-0.  I forgot this study of small schools designed as early college.  Also, note the comment I added clarifying the nature of my concern with the $6 million given by Gates to Fordham.]


Follow

Get every new post delivered to your Inbox.

Join 2,672 other followers