The Siren’s Call of Policy Influence

July 22, 2014

I’ve written several times before about education policy analysts who confuse the constant sound of their own voice for actual influence over policy.  There are many faux education experts who have never really done anything or studied anything that would support their self-proclaimed status as experts.  And the foundations that fund them are making a foolish mistake in thinking that the non-stop chattering of these faux experts actually influences anybody.  Education policymaking is a long game that requires investment in serious inquiry.  Solid evidence, not an insular circle of blabbering, moves the elite consensus and creates the conditions for enduring policy changes.

The problem is that not only are foundations remarkably under-funding serious inquiry, but the academics who should be engaging in that research are increasingly drawn to the siren’s call of policy influence.  Yes, it is a proper goal of policy research to have influence, but that influence is the end-product of serious work, not the thing for which the quality of one’s work and intellectual standards should be sacrificed.

The latest example of academics attempting to trade their integrity for influence can be found in Ray Fisman’s Slate article on vouchers in Sweden.  Fisman is a Harvard-trained economist who has rapidly risen to a named professorship at Columbia University’s business school.  He rose so rapidly because he has done some excellent work published in leading journals.

But even highly capable scholars have difficulty resisting the temptation to abandon their standards for their imagined ability to influence policy.  So, Fisman has also become a columnist for Slate.  His columns are nothing like his scholarship.  In particular, his recent piece on vouchers in Sweden was filled with glaring errors of fact as well as obvious flaws in causal reasoning.  Andrew Coulson has an excellent take-down of Fisman’s piece over at Education Next.  You should read Andrew’s entire piece, but some highlights of Fisman’s sloppiness include:

  • Fisman claims “more Swedish students go to privately run (and mostly for-profit) schools than in any other developed country on earth.”  In fact, only about 14% of Sweden’s students attend private schools, significantly less than the 68% in Belgium as well as higher numbers in a host of other developed countries.
  • Fisman’s only “evidence” that vouchers have harmed achievement in Sweden is that PISA scores have dropped in that country over the last decade.  The Razorback football program has also gotten a lot worse over the last decade.  By Fisman’s causal reasoning, perhaps Swedish vouchers are responsible for my poorly performing Hogs.  This is the type of mis-NAEP-ery we expect of Marc Tucker or Diane Ravitch, not a Harvard-trained economist in a named professorship at Columbia.

Fisman would never make such sloppy mistakes in a journal submission or conference presentation.  His colleagues would laugh at him.  But nothing seems to deter Fisman or other would-be Paul Krugmans from making laughable claims in the popular press.  Maybe academics should not be given such a free pass for whatever they write outside of journals.  Maybe the credibility of their scholarly work and their status within the academic community should also be called into question if they are willing to be so reckless.

Look, I know from personal experience the lure of policy influence.  I’ve been in the think-tank world and taken part in the silly collection of “metrics” of influence to get foundation funding.  And I’ve felt the temptation to claim expertise in areas beyond my scholarship.  But we all have to resist these temptations if we are to maintain the standards of academic work.  We need to maintain those standards so that research can remain credible and be the source of true, long-run policy influence.


July 22, 2014

(Guest Post by Patrick J. Wolf)

“Any other team wins the World Series, good for them. They’re drinking champagne, they get a ring. But if we win, on our budget, with this team… we’ll have changed the game. And that’s what I want. I want it to mean something.” – Billy Beane, Moneyball

The cost of baseball players once was largely ignored by the media and fans.  It was crude and destroyed the fun of the game, many thought, to inject hard-nosed considerations of efficiency into America’s pastime.  Then came Billy Beane’s “Moneyball” and the transformation of the Oakland A’s from perennial losers to a competitive and highly efficient professional baseball team, all due to careful consideration of how much bang players delivered for the buck.

Fast forward almost 25 years to the present day.  The cost of public education is largely ignored by both the media and education policymakers.  Many people think it is awkward, complicated or destroys the intrinsic and infinite worth of public education to inject hard-nosed considerations of efficiency into America’s schools.  Then came 2012, the first school year in history in which total U.S. government spending on public education went down.  Suddenly, money has to matter in public education, because apparently there isn’t an endless supply of it.

Today my colleagues and I at the School Choice Demonstration Project at the University of Arkansas released The Productivity of Public Charter SchoolsThe report is the first national study of the efficiency of charter schools relative to traditional public schools, and to tie funding to student achievement.  Across all 28 states in our study we found that public charter school sectors were more cost effective and/or generated a higher return on investment (ROI) than traditional public schools.  Public charter schools are like the Oakland A’s of public education — and last time I checked, the A’s had the best record in baseball.

Let’s start with cost effectiveness, or the amount of output generated per unit of input.   The charter school sectors in 21 states and D.C. all produce higher math and reading scores on the National Assessment of Educational Progress (NAEP) per $1,000 of per-pupil revenue than do the traditional public schools sectors in their respective states.  On average across our national sample, charters put up 17 more NAEP points in math and 16 more points in reading for every Cleveland they receive, which amounts to charters being 40 percent more cost effective in math and 41 percent more cost effective in reading.

pat post 1

What really matters, though, is how learning translates into future economic rewards for the student and society, commonly measured as ROI.  The charter sectors in 20 states and D.C. all outperform traditional public schools.  The weighted average ROI advantage from charter schooling across the national sample is almost 3 percent for a single year in a charter school and 19 percent if students spend half of their K-12 education in a charter.

Skeptics might say, “Charters gain their efficiency advantage by disproportionately admitting advantaged students.”  The data show this is not so.  A majority of the states in our sample have charter sectors that enroll a higher percentage of low-income students than their traditional public schools peers.  Although most of the charter sectors trail traditional public schools regarding the notoriously unreliable measure of special education enrollments, that gap is so small (3 percentage points) that it can’t plausibly explain the greater productivity of charters.  We used carefully matched samples of charter and traditional public school students from Stanford’s CREDO National Charter School Study to ensure that differences in student characteristics were unbiased.

pat post 2

Then what is behind the Money-Ed success of public charter schools?  Mathematically, the answer is simple.  Charters nationally are producing student achievement gains that are very similar to the levels in traditional public schools but receive about 30 percent less money per pupil.  Similar results at a lower cost explain the advantage for charters.  We can’t say for sure that charters would retain their productivity advantage over traditional public schools if they were funded on par with district-run schools, but it sure would be interesting to see what happens under something close to funding equity.  Instead of the Oakland A’s of the 2000s, playing competitively with 30 percent less payroll, equitably-funded charters might be the A’s of 2014, who look like world champions.

Common Core Political Naivete and the Enemies List

July 2, 2014

The entire Common Core enterprise has been characterized by shocking political naivete and over-reach.  Despite investing a fortune in political operatives and holding weekly conference calls “directed by Stefanie Sanford, who was in charge of policy and advocacy at the Gates Foundation,” the folks pushing Common Core did not anticipate that the Unions would betray them and oppose the implementation of Common Core as soon as it suited their purposes.  They did not anticipate that there was no authentic constituency for the proper implementation of the new standards and aligned high stakes tests.  They did not anticipate that the combined forces of the Unions and conservative opponents of centralized control would overwhelm the largely paid mercenaries they had on their side.  For people who imagine themselves politically sophisticated they look like a pack of amateurs.

And as the Common Core effort crumbles, its supporters are not just failing, but losing ground on previous accomplishments.   If you liked accountability testing, Common Core has done more to set back your efforts than Randi Weingarten ever could have done on her own.  As Rick Hanushek points out in the Wall Street Journal, the Unions are using Common Core not only to block new tests, but to eliminate high stakes testing altogether.  Several states will soon have no high stakes testing while they adopt a moratorium on stakes in their supposed transition to new tests.  The Gates Foundation has backed a two year delay in the hopes of rescuing their effort from collapse.  Like a retreating army suggesting a cease fire, they will find their opponents have little reason to keep the delay temporary.

In the hopes of achieving a total victory (changing standards and testing everywhere), the Common Core folks are going to end up with weaker testing and standards in many places.  As I suggested in my post on the Paradoxical Logic of Ed Reform Politics, seeking total victory often produces stunning defeat.

The other unintended side-effect of Common Core crumbling is that it is producing abusive efforts by its supporters to rescue it.  The whole enterprise depended on putting it into place quickly so that anyone who opposed the fait accompli could be dismissed as a kook or extremist.  The standards were adopted rapidly, but implementation of the high stakes tests has taken long enough for strong opposition to materialize.  Common Core may have captured Nijmegen, but the Arnhem of high stakes testing has proved a bridge too far.

This has not stopped the attempt to characterize opponents as kooks and extremists.  To be fair, some opponents are kooks and extremists, but many are not and Common Core supporters have had a bad habit of avoiding substantive debate by trying to dismiss their opponents as crazy.  There is something vaguely authoritarian about trying to centralize all education standards and testing, so not surprisingly Common Core supporters have also resorted to authoritarian tactics.  Taking a page from Tricky Dick, they have begun to use the power of the government to identify and punish opponents.

No, I’m not just talking about the threat that NCLB waivers and RTTP money would be more available to those who played ball with Common Core.  I’m talking about going after individuals who dissent.  Check out this story about  Brad McQueen, a teacher in Arizona, who published an op-ed against Common Core.

The state’s Associate Superintendent, Kathy Hrabluk, alerted her subordinates to this teacher’s dissent and asked them to “check your list of teacher teams (from which teachers are selected to work on tests at the Dept of Education)” so that he would not be involved in future teacher workgroups on state tests and other matters.  McQueen had been on those workgroups for the previous five years for which he received extra compensation.  No more.  As the Deputy Associate Superintendent for Assessments, Irene Hunting, replied to her boss, “We have made a note in his record.”  Another state official replied, “This was such a surprise for Arizona as Brad has been on many committees…  Let’s make sure he is not going to Denver later this month [to work on the new tests]. Please remove Brad McQueen from the list.”

Another Arizona education official, displaying all of the political sophistication of the Common Core movement, then replied on her government email, saying: “What a f*cktard.”

State education officials, doing their best to be the Common Core equivalent of the White House Plumbers, then proceeded to work on identifying one of McQueen’s fellow teachers to lend his or her name to a rebuttal op-ed that they would ghost write.  The bureaucrat in charge of PARCC for Arizona also called McQueen in his classroom to challenge him on why he opposed her test and quiz him about whether he was teaching the required standards.  McQueen feared they were fishing for grounds to terminate him and got off the call feeling like he has been threatened by a senior state official.

It’s an ugly story.  But this is what happens when you flirt with authoritarian reforms of education.  You start acting like an authoritarian.

(updated as described in comments)

It’s a Rookie Mistake

July 1, 2014

NCTQ has another report out ranking ed schools on whether they meet NCTQ’s ideas of what makes ed schools effective.  As I pointed out last year, NCTQ purports to have a strong research basis for claiming that ed schools should adhere to their standards, but that research is actually quite thin and often doesn’t support what NCTQ advocates.  I share NCTQ’s concern about improving the quality of teacher preparation, but I do not share their confidence that we know what works and certainly do not share their willingness to impose their preferences on everyone.  Unfortunately, we do not know the correct recipe for making better teachers even as NCTQ tries to make everyone cook the way they prefer.

Part of the advocacy campaign for NCTQ’s efforts is to lambaste ed schools for the fact that 1st year teachers tend to be less effective in the classroom as measured by valued-added on test scores.  According to the NCTQ narrative, if teachers do worse in their first year or two in the profession, it must be that ed schools are doing a lousy job of preparing them.  If ed schools were doing it correctly, there would be no negative effect for first year teaching.

In last year’s report NCTQ described how the shortcomings of novice teachers motivated their ranking system:

Should first-year teaching be the equivalent of fraternity hazing, an inevitable rite of passage? Is there no substitute for “on-the-job” training of novice teachers? The answers are obvious. We need more effective teacher preparation. Our profound belief that new teachers and our children deserve better from America’s preparation programs is the touchstone of this project.

And in pimping this year’s report, NCTQ’s tweeter feed repeats this same message: “If training & cert are mandatory, should be no reason to accept 1st yr as hazing ritual”  and “Novice struggle = struggle. Every year matters!”

This, of course, is a faulty argument.  Even when professionals are well-prepared, they may still improve with experience.  It is so widely recognized as a normal phenomenon that we even have a saying for people who are less good when they start — we say that they make “rookie mistakes.”  No one blames the minor leagues for the fact that big league rookies tend to be less effective.  No one denounces the Cavaliers for the fact that LeBron James got better with experience after moving to Miami.  It is normal for people to improve with experience, not necessarily evidence of their poor preparation.

But some see rookie mistakes as unacceptable in education because the stakes are too high.  Deborah Loewenberg Ball, the Dean of the Ed School at Michigan opines, with approving retweets from NCTQ, that: “Airline pilots don’t say, ‘My first few years of flying I was a wreck.’  That needs to be gone.”  We would never tolerate rookie mistakes among important professions, like airplane pilot or doctor.

In fact, we do tolerate rookie mistakes among doctors, pilots, and just about every profession.  A review of airline accidents reports that “inexperienced pilots have a 2-3 times increased incidence of mishaps due to pilot error.”  And this study of doctor errors in writing prescriptions found: “The overall detected error rate was 3.13 errors for each 1000 orders written…. First-year postgraduate residents were found to have a higher error rate (4.25 per 1000 orders) than other prescriber classes.”  In almost every profession there are returns on experience.  The striking thing about teaching is not that novice teachers are less effective, but that the improvement with experience is so small and basically flattens out by the third year.

All of us wish that doctors, pilots, teachers and other professionals would make no mistakes.  And we hope that improved training would reduce those errors.  But no matter how much NCTQ waves around the Flexner Report to justify its activities, teaching is not medicine and in teaching we do not have a scientific basis for saying how every teacher should be prepared.  NCTQ is not helped in its attempt to be the Flexner of education by mis-describing what research exists and by making sloppy errors of logic like claiming that the relative weakness of novice teachers is proof of poor teacher preparation.

These are the sorts of errors that people may be more likely to make without doctoral training and academic experience in the social sciences, which most of the staff at NCTQ and most other DC think tank/advocacy groups are lacking.  You might even call these rookie mistakes by novice researchers.

The Sweet Agony of Victory

June 30, 2014

(Guest Post by Matthew Ladner)

This has to be one of the most priceless photographs of all time- Faye Dunaway post Oscar victory, 1977. It will have to supplement


from now on.  Go score some victories so I can post Faye again soon.

Additional Thoughts on Vergara Decision

June 12, 2014

I don’t want to throw cold water on the excitement many folks have expressed abut the ruling in Vergara v. California ending teacher tenure protections, but I do think it is important to cool some of the heated enthusiasm.  Matt was appropriately cautious, noting that the decision will certainly be appealed and will take years to play out, but not everyone has been so measured.

I see the decision as more important as a symbol of the political challenges facing unions than as a change in policy that will significantly advance student achievement.  I’m skeptical of the educational impact of the ruling because:

1) It may well be reversed on appeal.  I’m also no lawyer, but I’m enough of a political analyst to see that the Courts are reluctant to make major policy changes without broad support from elites.  As Matt notes, the Courts tend to be lagging indicators of elite opinion, not cutting edge agents of change.  And this is as it should be.  We shouldn’t want unelected judges making too many big policy decisions without enough support from the democratically elected branches to ensure that the decisions can stand and be implemented.  It is an impressive sign of the fading political influence of teacher unions and the intellectual incoherence of some of their central policy positions that a judge was willing to strike down teacher tenure.  But I don’t think there is broad enough support for higher courts to stick with this policy stand.  They’ll find a way to walk back from the ledge.

2) Even if laws protecting tenure are struck down, it is unclear how broadly it will really be used to remove ineffective teachers.  There is a good amount of evidence that principals can distinguish between effective and ineffective teachers, but there isn’t a lot of evidence that principals will exercise that judgment very often even when they are empowered to do so.  Brian Jacob examined a program in Chicago that made it very easy for principals to dismiss teachers.  The good news is that when they dismissed teachers those teachers tended to be much less effective (as measured by VAM).  The bad news is that they rarely used their power to get rid of teachers.  Jacob wrote:

this analysis reveals that many principals – including those in some of the worst performing schools in the district – did not dismiss any teachers despite how easy it was under the new policy. This result is consistent with the fact that existing teacher contracts in many large urban school districts actually provide considerably more flexibility than is commonly believed and yet administrators rarely take advantage of such flexibility (Ballou 2000, Hess and Loup 2008, Price 2009). The apparent reluctance of many Chicago principals to utilize the additional flexibility granted under the new contract may indicate that issues such as teacher supply and/or social norms governing employment relations are more important factors than policymakers have realized.

To change those norms we need to address the motivation of principals to dismiss ineffective teachers even when they are empowered to do that.  Of course, when schools have to attract students and revenue in competitive systems, principals are more active in replacing teachers they deem ineffective.  Choice really does matter for other reforms to work well.


My former student and soon to be a professor of education, James Shuls, sent me his thoughts on Vergara.  Here is what he sent:


Back in April, I posted a series of quotes from Marcellus McRae’s closing argument in Vergara v. California to Jay Blog. Yesterday, the court handed down its decision and it appears that McRae was right, “You can’t make sense out of nonsense.”

Today, I have a piece on the Daily Caller summarizing the ruling and highlighting my take-a-way from the case.

On its face, this was a legal case that considered whether teacher tenure and other job protections violated California’s state constitution. At a more fundamental level, however, this was an evaluation of policies lauded by teachers unions throughout the country – teacher tenure, due process, and last-in, first-out provisions. For these policies to be found unconstitutional they first had to be proven to have an adverse effect on disadvantaged students; and indeed, they were.

I go on to say:

Legally, there are still many questions to be resolved. In the court of public opinion, however, the ruling could not be clearer: Teacher tenure has been tried and it has been found wanting. You simply cannot make sense out of nonsense.

I invite you to check out the full piece here.


James Shuls is the Director of Education Policy at the Show-Me Institute. Follow on Twitter @shulsie

Checking Privilege is Bad Politics

May 27, 2014

Recently I described the political advantage of choice over top-down reforms.  Choice creates its own constituency to protect and expand it because people will fight to keep choice once they have it.  Top-down reforms, by contrast,  are the most popular on the day they are adopted and decline after that, leaving them vulnerable to being blocked, diluted, co-opted, or repealed.  Who will protect and expand a system that imposes consequences for test performance?  The people who are punished by it know who they are and are well organized.  The beneficiaries (if any) are dispersed and disinterested.  Where is the “test our kids more rally” being held?  Nowhere.

It’s a basic political science insight that many well-financed education reformers somehow lack — concentrated and organized interested tend to prevail over dispersed and unorganized interests.  Choice is consistent with this basic lesson while top-down reform runs contrary to it.

An important corollary of this basic lesson is that people with more money tend to be better organized and effective at protecting their interests than poor people.  So, designing a program to stick it to wealthy people is generally a bad idea.  If you are pushing for the expansion of choice, don’t exclude wealthy people.  A s the old saying goes, “Programs targeted for the poor tend to be poor programs.”  If wealthy people are included among those who can benefit from a choice program, they can organize to protect and expand that program.  If a choice program only offers benefits to the most disadvantaged, those beneficiaries are not well-positioned to fight for it politically.  You need to include more advantaged people as beneficiaries so they can fight for a program that also benefits the poor.

If you need an example of the political logic of how universal programs are actually more effective at helping the poor than targeted programs, just compare Social Security and WIC.  Social Security is generous, paying recipients much more on average than they paid in.  It pays cash which recipients can use in any way they want.  It comes automatically; you don’t have to wait in a long line in a dank office to apply to a surly bureaucrat to get it.  It is also indexed to inflation, so it never loses value over time. And — most importantly — it is extremely effective at alleviating poverty among seniors.

WIC, on the other hand, provides meager food assistance to low income families.  It can only be used for certain foods and not other things that poor families might want.  This jerk with a blog in the New York Times is actually outraged that WIC might change its rules to allow poor families to buy white potatoes:

I have nothing against potatoes, either. But there’s almost no one in America, WIC recipients included, who isn’t getting enough potatoes. And that’s what the Institute of Medicine (I.O.M.) was thinking when it excluded potatoes from the WIC program. Because everyone knows that in the United States, “potatoes” equals “fries.”

And if it isn’t degrading enough to be told what to eat by Mr. Bossy Pants in the New York Times, you have to wait in long  lines and engage in endless self-disclosure in forms before you can get WIC assistance at all.

Because Social Security is universal in its benefits — the checks go to the rich and poor alike — the program is politically very well protected, long enduring, and provides fantastic benefits.  Because it targets the poor, programs like WIC are always politically vulnerable, are constantly being replaced (remember AFDC?), and provide lousy benefits.  If you don’t want your education reform to look like WIC, don’t exclusively target the poor.

Even when choice programs target their benefits to the poor, they usually have the good sense not to take something away from the rich.  Rich suburbanites are no worse off if poor kids in Milwaukee have some extra choice.  Of course, the program would be less politically vulnerable, provide more generous benefits, and would be even larger if it also offered benefits to people with more money.

But top-down reforms often take something important away from wealthy families — control over their child’s education.  When a wealthy suburban mom wants her child taught standard algorithms for math in 2nd grade, she doesn’t want to be told that Common Core requires that those algorithms not be introduced until 4th grade.   Even if Common Core actually requires no such thing, the fact that the local district tells her that there is nothing that she or they can do about it, makes that mom feel like she has no control and no recourse.  At least if she were told something like this in the past, she would know which school board member to call or which state legislator to mobilize in her defense.  But to whom does she complain to change what is required by Common Core (or what is alleged to be required by Common Core — whether it really is or not makes no difference)?

Wealthy moms also tend not to like their children being given a bunch of dumb tests and told (again, perhaps wrongly) by their school that they can’t learn more interesting and diverse material because test-based reforms require it.  They especially get annoyed when they believe that their child could pass the test regardless of what is taught.  So they see virtually no benefit from test-based reforms and see significant impingement on control over their child’s education.

It does no good for defenders of top-down reforms to complain about “white suburban moms” as Education Secretary Arne Duncan did.  You can’t guilt wealthy folks out of wanting to protect what they believe is in the best interests of their children.  And it may feel good to self-absorbed reformers to declare that they are pushing top-down reforms to check white privilege, but it is lousy politics.  If your goal is to actually do something to help poor people rather than feeling righteous about sticking it to the wealthy, avoid top-down reforms and push universal choice instead.


Get every new post delivered to your Inbox.

Join 2,538 other followers