Peer Review, DC Style

September 16, 2009


Elwood Dowd admires a painting of himself with the DOE peer reviewer

(Guest post by Greg Forster)

Eduwonk reports that Arne Duncan and his Race to the Top team are discovering just one of the many fun flaws of “peer review”:

A lot of behind the scenes chatter and concern and I’d say even worry that it’s going to be hard to get “Race to the Top” proposal peer reviewers who know a lot about school reform – and proposals like this are complicated.  There are a lot of conflicts among the usual suspects.  After all, teachers’ unions have to sign on off the applications and can benefit from them so they’re self-interested, most wonks outside of government are helping various states get together ideas and applications, and states themselves are pretty self-interested, obviously.  Add on to that the generally meager rewards of peer reviewing in the first place and this issue has a lot of folks chattering about exactly who can do this work in a high-quality way…

So it sounds like the standard they’ve set for themselves is to find reviewers who know a lot about school reform but have no vested interest in school reform. How many people did they think were going to fall into that category?

Ben Stein in Ferris Bueller (wide)

Anyone? . . . Anyone?

Maybe “peer review” wasn’t really an appropriate rubric for evaluating government grant proposals. Has anyone ever suggested, say, peer review for Pentagon contracts?

But then, if they don’t put some kind of academic-sounding veneer on it, the thoroughly politicized nature of the process will be too embarrassingly transparent.

Of course, when they do “peer review” in academia they have the best of both worlds. The process is just as corrupted by the self-interest of the participants – mostly not in terms of politics but in terms of their desire to promote research that agrees with their own findings and suppress research that might call their own findings into doubt – and yet because the reviewers are professional academics everything is assumed to be done in the interests of scholarship.

Still More Lefties for School Choice

September 9, 2009

Monopoly - Pennybags

Can you sell school choice to monopolists?

(Guest post by Greg Forster)

We’ve been tracking the increasing movement on the political left toward school choice.

Now, Arianna Huffington repackages school choice for the lefty crowd, connecting it to the health care debate by calling choice “single payer for education,” i.e. you choose the provider and government pays.

The association is distasteful, given that the pending government takeover of healthcare is a knife at the throat of our freedom. But if this is how some people need to think about it in order to see the point about school choice, that’s fine as far as it goes – in education, moving to a “single payer” system would be a step in the right direction, as opposed to a step in the wrong direction as with health care.

HT Eduwonk

No Jack Jennings Is Not on Fire

July 29, 2009

No two people are not on fire

(Guest post by Greg Forster)

Stop the press! How did I miss this on Eduwonk last week?

At this point if Jack Jennings doused himself in gasoline and set himself ablaze in front of the NEA, would anyone notice?

Hey, that’s what happens when you spend too long peddling political hackery trumped up as research. Sooner or later, people get wise to the con and stop taking you seriously.

Of course, Andy feels the need to call Jennings’ work “important.” But if all the empty, generic words of praise people rotely intone about Jennings doused themselves in gasoline and set themselves ablaze in front of the Merriam-Webster publishing comany, would anyone notice?

In other Eduwonk news, give Andy credit for not drinking too much of the yesterday’s new Race to the Top flavor Kool-Aid; he linked to this item, which helps illustrate just how deep the kabuki goes.

The National Standards Sausage-Making

June 9, 2009

Every decade or so we have to debate the desirability of adopting national standards for education.  People tend to be in favor of them when they imagine that they are the ones writing the standards.  But when everyone gets into the sausage-making that characterizes policy formulation, it generally becomes clear that no one is going to get what they want out of national standards.  What’s worse is that the resulting mess would be imposed on everyone.  There’d be no more laboratory of the states, just uniform banality.

Of course, some people always hope that they’ll somehow manage to sneak their preferred vision into place without having to go through the meat grinder.  That’s what is happening now with the National Governor’s Association effort at “voluntary” national standards.  In a process completely lacking in transparency and open-debate, some are rushing to announce a national standards fait accompli.

My colleague Sandra Stotsky tells us what’s what:

“If another country wanted other countries to respect its educational system and the reforms it was trying to make, who would it choose to lead such an important professional project as the development of its national standards in mathematics and in the language of its educational system itself?  In any other country in the world, one would expect a distinguished mathematician at the college level to be asked to chair the mathematics standards-writing committee–someone who commands the respect of the mathematics profession (and obviously is an expert on mathematics).   For the language standards-writing committee, one would likewise expect an eminent scholar in a college-level department–someone whose command of the language and understanding of the texts that inform the development of this language could not be questioned.   If the National Governors Association and the Council of Chief State School Officers had thought about national pride (and national need) as well as academic/educational expertise, then all of us would respect the Common Core Initiative and look forward with eagerness to the drafts the NGA and CCSSO have promised to make public in July.

 These two organizations could have followed, for example, the exemplary procedures followed by the National Mathematics Advisory Panel, on which I had the privilege to serve.  The Panel was chaired by the former president of one of the major universities in the country, all Panel members were identified at the outset, their qualifications were made known to the pubic, their procedures were open to the public and taped as well, and the final product was hammered out in public, after dozens of reviewers provided critical comments. 

 But instead of choosing nationally known scholars to chair and staff these committees–to assure us of the integrity and quality of the product–the NGA and the CCSSO have, for reasons best known to themselves,  treated the initiative as a private game of their own.  The NGA and the CCSSO haven’t even bothered to inform the public who is chairing these committees, who is on them, why they were chosen, what their credentials are, and why we should have any confidence whatsoever in what they come up with. 

 One person has announced on his own to the press and to a state department of education that he is chairing the mathematics standards-writing committee. He has not been contradicted by anyone at NGA or CCSSO, so we must assume he’s for real.  It turns out he is an English major with no academic degrees in mathematics whatsoever.  No one has yet announced on his/her own that he/she is chairing the English standards-writing committee.   One wag has already wondered whether this person might be a mathematics major with no academic degrees in English.  But it’s possible the sad joke in mathematics is not being repeated in English. 

 This country deserved better for a project of such national importance.”

Sandy Kress added these words of wisdom (pardon the capitalization since this was a comment on a post at Eduwonk):

“i suspect after the good feelings wear off, other governors and chiefs will begin to ask whether they can or should consider new standards at this time. once they learn about how hard it is to write new standards, they will ask even more questions. when we get to the controversies around whole language vs. phonics, they will ask more questions still. then comes computation vs. concepts. then comes all the many questions that arise once you get below the level of 30,000 feet. then – God forbid – you might even get to the place where you might possibly find the new standards under consideration to be no better than (or even possibly worse than) the standards you have! could it be that the tradeoffs that happen nationally will be the same as those that occur in the states? could the same interest groups intervene? could this nice dream be interrupted by the demons that bedevil state standard setting? could these interests be the problem as much as variation? oh no, could it be there’s no santa… no, i won’t go there.

and, oh yes, what about performance standards? if we ever get to detailed precise standards in each grade for reading and math, do the participants agree to common performance standards? if they don’t, who’s kidding whom? the real problem today is not so much that some states have vastly higher standards than other states; it’s more that their performance standards are greatly different. have the states, or will the states, commit to making those the same? if not, this will be utterly fruitless.

listen – DO NOT GET ME WRONG – i’m all for higher, fewer, clearer standards. i’ve spent a lot of time working on improving texas’ standards over the past 20 years. i’ve spent a lot of time with the hunt institute pushing more common standards. this is indeed the right thing to do.

but this process is going to be much more difficult than some think. it won’t happen overnight, nor should it. and there will remain great variation at the end of the day. it is utterly naiive and/or foolish to expect states to jump track from their current gameplans, particularly where they’re reasonably well thought out.

be prepared for states to recognize this “the morning after.” texas just recognized it before “the drinking began.”

also be prepared to realize that a better approach might be for one or more of these organizations to begin by recruiting the best and the brightest and actually doing the hard work of developing a few sets of model standards and then shopping them to the states, with the political support of those who rightly want high, common standards as well as perhaps some incentives from the feds to take these steps.” 

(edited for typos)

Beltway Confusion

March 3, 2009

(Beltway edu-analysts discuss the world over brandy and cigars.  Note where they are headed.)

I feel sorry for my education colleagues within the DC Beltway.  I don’t know if it’s all those wine and cheese receptions or box lunch lectures that addle their brains, but they are clearly confused.  They confuse political analysis for research.  And they confuse their political preferences for political analysis. 

Look, for example, at the recent post by my friend Andy Rotherham at Eduwonk on vouchers, which states:

“Now, paradoxically, the school choice experience since the early 1990s has lessened the allure of vouchers as a scalable education reform but at the same time made these smaller “pilot” type initiatives like the one in D.C. seem less toxic and more harmless among an increasing number of players.   Opponents don’t even really have a slippery slope to point to in any of the early adopter sites for vouchers.  There’s not one in D.C.  There it’s the public charters not the vouchers that are taking over and not in the other cities/states, even Milwaukee, where vouchers have been tried and the effects have been modest.  In other words, vouchers are not destroying the public schools.  Rather, systemically, they’re not really doing much of anything at all.”

Andy concludes that “systemically” vouchers aren’t “much of anything at all” because they aren’t expanding very rapidly.  This is a political analysis on the appeal of larger voucher programs, not a summary of research on the effects of vouchers on public school achievement.  If Andy had wanted to talk about the research on the systemic effects of vouchers he would have referenced this literature, which clearly shows that expanding choice and competition through vouchers improves public school performance.  So, Andy substitutes political analysis for research.

But he also substitutes his political preferences for political analysis because he ignores the steady growth in vouchers over the last two decades.  There are now 24 voucher or tax-credit programs in 15 states serving over 100,000 students.  Just last year two new programs were adopted, in Georgia and Louisiana, and the tax credit program in Florida was significantly expanded.  Andy may wish the voucher movement to be stalled, but a clear political analysis would reveal that vouchers continue to move forward.

Now, it’s true that there have been setbacks in the voucher movement.  And it’s true that each new program encounters a blizzard of opposition, making each step forward seem inordinately difficult.  But vouchers are just the spearhead of a broader choice movement that includes the more rapidly expanding charter movement.  If not for the viability of voucher programs, charters would have been the target of this onslaught of opposition. 

Vouchers have made the world safe for charters.  And the moment that vouchers really do stall, the enemies of school choice will redirect their fire at charters, strangling them with regulation and repealing charter gains.  To say that vouchers haven’t really done much of anything politically because charters are really where the action is ignores how much charters owe their political strength to the credible threat of new and expanded voucher programs.

It may be fashionable at Beltway receptions to dismiss vouchers as everyone is eager to be seen as championing the latest DC fad proposal.  But real analysis of research and politics show that the expansion of school choice, led by vouchers, will have a greater impact on education reform than building new school buildings, expanding pre-school, adopting a 21st century curriculum or whatever folks there are now talking about.

(edited for typos)

AFT and UAW – More Alike Than You’d Think

December 30, 2008

aft uaw1

(Guest post by Greg Forster)

Lots of people are picking up on the temper tantrum about alleged “demonizing of teachers” begun by a Randi Weingarten speech and continued in Bob Herbert’s column on the speech.

Even that notorious right-winger Eduwonk points out that Weingarten and Herbert are hitting a straw man. I think the real problem is not that school reformers demonize teachers but that defenders of the government school monopoly angelize them. When we reformers insist that teachers should be treated as, you know, human beings, who respond to incentives and all that, rather than as some sort of perfect angelic beings who would never ever allow things like absolute job protection to affect their performance, it drives people like Weingarten and Herbert nuts.


A typical teacher, as seen by Randi Weingarten

But what I’d like to pick up on is the question of whether the troubles of the government school system are comparable to the troubles of the auto industry.

Of the alleged demonizing of teachers, Herbert had written:

It reminded me of the way autoworkers have been vilified and blamed by so many for the problems plaguing the Big Three automakers.

Eduwonk points out Herbert’s hypocrisy (though he delicately avoids using that word) on this point, because elsewhere in the column, Herbert praises Weingarten for expressing a willingness to make concessions on issues like tenure and pay scales. Union recalcitrance on these types of reform, Eduwonk points out, is precisely why the auto industry is in so much trouble, and Weingarten has been driven to make noises in favor of reform because a similar dynamic has been at work in the government school system.

On the other hand, Joanne Jacobs thinks the comparison between the AFT and the UAW is inapt:

 I don’t think skilled teachers and unskilled auto workers have much in common.  Auto unions pushed up costs, especially for retirees, making U.S. cars uncompetitive.  In education, the problem isn’t excessive pay, it’s the fact that salaries aren’t linked to teacher effectiveness, the difficulty of their jobs or the market demand for their skills.

But teachers’ unions have pushed up costs – dramatically. In the past 40 years, the cost of the government school system per student has much more than doubled (even after inflation) while outcomes are flat across the board. And this has mainly been caused by a dramatic increase in the number of teachers hired per student – a policy that benefits only the unions.

It’s true that high salaries aren’t the main issue in schools, although teacher salaries are in fact surprisingly high. The disconnect between teacher pay and teacher performance is much more important. But the UAW has the same problem! Their pay scales don’t reward performance, either.

The source of Jacobs’ confusion is her mistaken view that auto workers are “unskilled.” Farm workers are unskilled, but not auto workers. The distinction she’s reaching for is the one between white-collar or “professional” work and blue-collar work. But some blue-collar work is skilled and some is unskilled, and auto workers are in the former category. This matters because with skilled blue-collar workers, as with white-collar workers, there’s a dramatic increase in the importance of incentives as compared with unskilled labor.

In fact, a lot of smart people have been arguing (scroll down to the Dec. 26 post) that exorbitant salaries and benefits aren’t nearly as much of a problem in the auto industry as union work rules – including poor performance due to absolute job protection, pay scales that don’t reward performance, and rigid job descriptions that make process modernization impossible.

Sound familiar?


Bloggers Shouldn’t Have Rapper Names

October 8, 2008

In my last post I described Jennifer Jennings as the blogger formerly known as Eduwonkette.  I had thought we only had to call her Eduwonkette when we didn’t know who she was.  But I guess she continues to go by her rapper name, Eduwonkette.  And Aaron Pallas, an otherwise respectable scholar, continues to call himself Skoolboy — with a k!  And I guess they are both cribbing (in the non-rapper meaning) from Eduwonk, who we’ve always known to be Andy Rotherham.

I find the use of rapper names by bloggers to be downright silly.  It’s especially silly when accompanied by self-aggrandizing cartoons and graphics.  Here at Jay P. Greene’s Blog we’ve gone for a minimalist approach, both out of laziness and an aesthetic vision that tried to put the focus on content.

But if a bunch of other folks are going to continue to call themselves rapper names and have cartoon graphics to represent themselves, maybe I should do the same.  Perhaps I should go by my rapper name — DJ Super-Awesome.  And maybe we should use Thundarr the Barbarian graphics to represent ourselves.  I call the image of Ookla and I’ll let Greg and Matt fight over who gets to be Ariel.

Eduwonk: Vouchers Boring, Bus Service Consolidation Fascinating

July 23, 2008

(Guest post by Greg Forster)

One of Eduwonk’s most important contributions to the education blogosphere is the snarky comment. Snark is frowned upon in these parts (don’t leave a snarky comment on this post, or you’ll be violating site policy). But if I may register a minority report on behalf of the loyal opposition, I think snarky comments do for the marketplace of ideas what short sellers and speculators do for the stock market: they’re not classy or respectable, but they perform an indispensable price-discovery function. (Another similarity is that the people who do them make tons of money – right, Andy? Andy?)

Of course, not all brokerage houses are interested in having their services used by short sellers, and there may be some sense in declaring some education blogs as snark-free zones. The short sellers can always take their business to other brokers – just like I post all my snarky comments on Pajamas Media.

But if Eduwonk is going to dish out snark, he’d better be able to take it. So check out what I found while scrolling through Eduwonk this morning.

In response to John McCain’s big education speech at the NAACP:

If this campaign turns into a debate about vouchers please just shoot me now. I’d prefer a debate that ignores education than that tired fight again.

Two days earlier:

Shouldn’t we be . . . trying to get school districts out of the busing business altogether? Big school districts like to boast about how they bus more passengers each day than Greyhound. That’s true, but also sort of insane if you think about it and consider that their primary mission is teaching and learning. Besides, today’s buses are horrendous polluters even when greener technology is available, control over transportation means control over parental decision-making, and school districts often aren’t even very good at designing efficient transportation schemes or adapting to changing circumstances like $4 gas, which was not exactly an unforeseen issue in the transportation world…Student safety means that, especially for younger students you want to be careful about how you merge transportation schemes, but having local or regional agencies that handle transportation would pay a lot of dividends if was approached with the dual principles of being greener and more parent- and civic-friendly at the front-end.

Got that? A debate about the policy that represents the most fundamental break from the existing system, is most consistently supported by empirical evidence, and is currently the most politically successful movement in the education world is boring enough to induce suicide. But the prospect of transferring control of bus services from school districts to local or regional transportation agencies is fascinating.

Supply your own snarky comment; I’m not allowed.

But before leaving the subject, I will note that if you’ve seen the new Batman movie, you already know whom to call for all your school bus operation needs:

You know how I got these scars? I was in an accident caused by an incompetent school bus driver, because educational transportation is controlled by school districts whose core mission is teaching and learning rather than by local or regional transportation agencies!

Educating Journalists about Education Science

July 16, 2008

(Guest post by Greg Forster)

Don’t worry, this post is definitely not a continuation of the recent big dustup about 1) whether it’s naughty for scholars to provide journalists with accurate information about their work; and 2) whether it’s naughty for anonymous bloggers to argue that scholars’ motives are relelvant to their credibility, but bloggers’ motives aren’t relevant to theirs (which reminds me of Pat Moynihan’s quip about the Supreme Court cases, since overturned, holding that government can’t subsidize private school books but can subsidize classroom equipment such as maps; Moynihan asked, “What about atlases?” – books of maps? What about scholars who are bloggers? Or bloggers who write about scholarly studies? Once you start legitimizing ad hominem arguments, where do you stop?).

But I would like to expand on a comment that Eduwonk made during said dustup, which deserves more attention and has significance well beyond the issues that were at stake in that squabble. The comment got lost in the exchange because it was somewhat tangential to the main points of contention.

He wrote:

Not infrequently newspapers get snookered on research and most consumers of this information lack the technical skills to evaluate much of the work for themselves.   As education research has become more quantitative — a good thing — it’s also become less accessible and there is, I’d argue, more an asymmetry to the information market out there than a fully functioning marketplace of ideas right now.  In terms of remedies there is no substitute for smart consumption of information and research, but we’re not there yet as a field.

We are living in the first golden age of education research, brought on by the advent of systematic data collection, which every other field of human endeavor began undertaking a long time ago but which education is only getting around to now because it has been shielded from pressure to improve thanks to its protected government monopoly. Given the explosion of new information that’s becoming available, educating journalists about quantitative research is a huge problem. Jay is right that there is a marketplace of ideas. There really can’t help but be one; the idea some people seem to have that we can forbid people who own information from spreading it around as much as they want is silly. But just because there’s a market doesn’t mean there’s a perfect market, and Eduwonk is right that markets require informed consumers to function well. The current state of methodological ignorance among journalists does hinder the market of ideas from functioning as well as it should. (I’ll bet Jay would agree.)

As it happens, the same subject came up this morning in a completely different context, as my co-workers and I struggled to figure out the best way to present the findings of an empirical study we’re coming out with so that journalists will be able to follow them. And I wasn’t there, but I hear this topic also came up at a bloggers’ panel at the recent conference of the Education Writers’ Association.

Here at the Friedman Foundation, this has been a topic of great importance to us for some time, since exposing the bad and even bogus research that’s used to justify the status quo is one of our perennial challenges. We took a stab at composing a journalist’s guide to research methods. It went over well when we first distributed it (at last year’s EWA, if memory serves). But it’s necessarily very basic stuff.

Eduwonk is also right about journalists having been snookered by lousy research, and I think that has had both good and bad effects. The good news is that I’ve noticed a clear trend toward greater care in reporting the results of studies (not at propaganda factories like the New York Times, of course, but at serious newspapers). In particular, we’re seeing journalists talk about studies in the context of previous studies that have looked at the same question. Of course, we have a long way to go. But we’re on the way up.

On the bad side, however, I have also noticed a greater reluctance to cover studies at all. Part of that is no doubt due to the increase in volume. I’m young, but even I can remember the heady days of 2003 when any serious empirical study on the effects of a controversial education policy (vouchers, charters, high-stakes testing) would get at least some coverage. Now it’s different, and (to echo Eduwonk) that’s a good thing. But I think it’s extremely unlikely that this is the only factor at work. Junk science has poisoned the well for serious research. No doubt that was part of its intended purpose (although of course the motives of those who produce it have no relevance to its scientific merts or lack thereof).

My hope is that journalists will soon realize they’re getting left behind if they don’t learn how to cover the research accurately. Their job is to go where the news is. If the news is in quantitative research – and that is in fact where a lot of it is – they’ll have to learn how to get there.

Also, the changing media landscape will help. The old idea that journalists must be neutral stenographers with Olympian detachment from all the issues they cover is an artifact of the mid-20th-century role of the media as oligarchic gatekeeper, and is rapidly dying out. As “news” increasingly includes coverage by people who are actively engaged in a field, even as advocates, we can expect the news to be increasingly provided by people with greater amounts of specialized knowledge. (By the way, the old idea of the scholar as detached Olympian stenographer is equally an artifact of vanished circumstances, and will probably be the next thing to go; see the Our Challenge to You statement on the inside cover of any empirical study published by the Friedman Foundation for our views on the relationship between advocacy and scholarship.)

An optimistic view, yes – but since my optimism on other subjects has been triumphantly vindicated over the past year, even when the conventional wisdom said to head for the hills, I think I’ll let it ride.

Blog Rankings

July 14, 2008

This blog is not yet three months old but I am pleased to report that it is off to a good start.  According to Technorati’s rankings, is attracting more readers than the American Federation of Teachers’ blog, Edwize, more than Diane Ravitch and Deborah Meier’s, Bridging Differences hosted by Education Week, more than the Reason Foundation’s Out of Control, and the Center for Education Reform’s Edspresso.  It significantly trails the educouple of Eduwonk and Eduwonkette as well as Cato at Liberty (although that’s not primarily an education blog).  Flypaper, which started about the same time as this blog, is also off to a good start.  The Queen of education blogs seems to be Joanne Jacobs.

Here are the Technorati rankings (as of this morning) of education sites that seem to share some of the same audience as this blog.  By no means is this a comprehensive list of education blogs.  And I have no idea how reliable or meaningful Technorati’s rankings really are.  I’d continue blogging no matter what the rankings were because it’s fun.  I imagine the same is true of most others.

  1. Cato at Liberty               3,662
  2. Joanne Jacobs                3,709
  3. Eduwonkette                27,419
  4. Eduwonk                      30,876
  5. Flypaper                       95,943
  6. Jay P. Greene               104,227
  7. Bridging Differences   107,924
  8. D-Ed Reckoning         107,924
  9. AFT’s Edwize              116,227
  10. Edspresso                  123,039
  11. Out of Control            123,039
  12. Core Knowledge         127,851
  13. Sherman Dorn            151,703
  14. EdBizBuzz                   184,730