Surprise! What Researchers Don’t Know about Florida’s Vouchers

April 21, 2008

(Guest post by Greg Forster)

 

Florida’s A+ program, with its famous voucher component, has been studied to death. Everybody finds that the A+ program has produced major improvements in failing public schools, and among those who have tried to separate the effect of the vouchers from other possible impacts of the program, everybody finds that the vouchers have a positive impact. At this point our understanding of the impact of A+ vouchers ought to be pretty well-formed.

 

But guess what? None of the big empirical studies on the A+ program has looked at the program’s impact after 2002-03. That was the year in which large numbers of students became eligible for vouchers for the first time, so it’s natural that a lot of research would be done on the impact of the program in that year. Still, you would think somebody out there would be interested in finding out, say, whether the program continued to produce gains in subsequent years. In particular, you’d think people would be interested in finding out whether the program produced gains in 2006-07, the first school year after the Florida Supreme Court struck down the voucher program in a decision that quickly became notorious for its numerous false assumptions, internal inconsistencies, factually inaccurate assertions and logical fallacies.

 

Yet as far as I can tell, nobody has done any research on the impact of the A+ program after 2002-03. Oh, there’s a study that tracked the schools that were voucher-eligible in 2002-03 to see whether the gains made in those schools were sustained over time. But that gives us no information about whether the A+ program continued to produce improvements in other schools that were designated as failing in later years. For some reason, nobody seems to have looked at the crucial question of how vouchers impacted Florida public schools after 2002-03.

 

[format=shameless self-promotion]

 

That is, until now! I recently conducted a study that examines the impact of Florida’s A+ program separately in every school year from 2001-02 through 2006-07. I found that the program produced moderate gains in failing Florida public schools in 2001-02, before large numbers of students were eligible for vouchers; big gains in 2002-03, when large numbers of students first became eligible for vouchers; significantly smaller but still healthy gains from 2003-04 through 2005-06, when artificial obstacles to participation blocked many parents from using the vouchers; and only moderate gains (smaller even than the ones in 2001-02) after the vouchers were removed in 2006-07.

 

[end format=shameless self-promotion]

 

It seems to me that this is even stronger evidence than was provided by previous studies that the public school gains from the A+ program were largely driven by the healthy competitive incentives provided by vouchers. The A+ program did not undergo significant changes from year to year between 2001-02 and 2006-07 that would explain the dramatic swings in the size of the effect – except for the vouchers. In each year, the positive effects of the A+ program track the status of vouchers in the program. If the improvements in failing public schools are not primarily from vouchers, what’s the alternative explanation for these results?

 

 

 

 

Obviously the most newsworthy finding is that the A+ program is producing much smaller gains now that the vouchers are gone. But we should also look more closely at the finding that the program produced smaller (though still quite substantial) gains in 2003-04 through 2005-06 than it did in 2002-03.

 

As I have indicated, I think the most plausible explanation is the reduced participation rates for vouchers during those years, attributable to the many unnecessary obstacles that were placed in the path of parents wishing to use the vouchers. (These obstacles are detailed in the study; I won’t summarize them here so that your curiosity will drive you to go read the study.) While the mere presence of a voucher program might be expected to produce at least some gains – except where voucher competition is undermined by perverse incentives arising from bribery built into the program, as in the D.C. voucher – it appears that public schools may be more responsive to programs with higher participation levels.

 

There’s a lot that could be said about this, but the thing that jumps to my mind is this: if participation rates do drive greater improvements in public schools, we can reasonably expect that once we have universal vouchers, the public school gains will be dramatically larger than anything we’re getting from the restricted voucher programs we have now.

 

One more question that deserves to be raised: how come nobody else bothered to look at the impact of the A+ program after 2002-03 until now? We should have known a long time ago that the huge improvements we saw in that year got smaller in subsequent years.

 

It might, for example, have caused Rajashri Chakrabarti to modify her conclusion in this study that failing-schools vouchers can be expected to produce bigger improvements in public schools than broader vouchers. In this context it is relevant to point out that many of the obstacles that blocked Florida parents from using the vouchers arose from the failing-schools design of the program. Chakrabarti does great work, but the failing-schools model introduces a lot of problems that will generally keep participation levels low even when the program isn’t being actively sabotaged by the state department of education. If participation levels do affect the magnitude of the public school benefit from vouchers, then the failing-schools model isn’t so promising after all.

 

So why didn’t we know this? I don’t know, but I’ll offer a plausible (and conveniently non-falsifiable) theory. The latest statistical fad is regression discontinuity, and if you’re going to do regression discontinuity in Florida, 2002-03 is the year to do it. And everybody wants to do regression discontinuity these days. It’s cutting-edge; it’s the avant-garde. It’s like smearing a picture of the virgin Mary with elephant dung – except with math.

 

You see the problem? It’s like the old joke about the guy who drops his keys in one place but looks for them in another place because the light is better there. I think the stats profession is constantly in danger of neglecting good research on urgent questions simply because it doesn’t use the latest popular technique.

 

I don’t want to overstate the case. Obviously the studies that look at the impact of the A+ program in 2002-03 are producing real and very valuable knowledge, unlike the guy looking for his keys under the street lamp (to say nothing of the elephant dung). But is that the only knowledge worth having?

 

(Edited to fix a typo and a link.)

3 Comments | competitive effects, methodology, research reports, vouchers | Tagged: , , , | Permalink
Posted by Greg Forster

More Money Myth

April 20, 2008

An article in today’s LA Times illustrates how the money myth is alive and well.  The piece by Seema Mehta focuses on private fund-raising efforts in California that are seeking to off-set proposed budget cuts. 

The article, and the people quoted in it, wish to establish 1) that California spends far too little on education, which is demonstrated by the alleged fact that it spends less per pupil than almost all other states; 2) that private fund-raising is necessary to make a significant difference in remedying those perceived shortfalls; and 3) that inequities in the capacity of different communities to engage in private fund-raising is a significant contributor to inequities in student achievement between those communities. 

All three of these claims are inconsistent with the available evidence. Mehta attempts to establish the first claim that California spends far less than most states by asserting, “The state ranks 46th in the nation in per-pupil spending.”  According to the U.S. Department of Education’s most recent Digest of Education Statistics, total per pupil spending in California ranked 23rd of the 51 states and DC, not 46th.  Total per pupil spending was $9,655, trailing the national average of $10,071, but not by much.  It’s true that the cost of living is higher in California.  Perhaps it would be desirable for California to spend more.  But the claim that California woefully under-spends on education would have to be supported by systematic evidence, none of which is provided in the article — other than the false ranking.

The second claim that private fund-raising is an essential part of overcoming budget shortfalls is also inconsistent with the evidence.  In a chapter I wrote for Rick Hess’s book on education philanthropy, With the Best of Intentions, I found that total private giving to public education is a tiny portion of total spending on schools.  All giving, from the bake sale to the Gates Foundation, makes up less than one-third of 1% of total spending.  It’s basically rounding error.  This is not to say that private giving to public schools can’t do some good.  It’s just completely unrealistic to expect private funding to make-up for or significantly supplement public funding.  The taxing power of the government generates over half a trillion dollars each year for public education, which would entirely consume the net worth of the 12 richest people in the world in a single year. 

But the LA Times article suggests that private giving can (and must) make a big difference.  It cites the example of the Irvine Public Schools, which receives $3 million annually from a community foundation.  it also quotes the head of that foundation saying, “The only way to take good districts and make them great is to do private fund-raising. But it’s even more urgent now with the terrible budget cuts.”  Nowhere does the article mention that this $3 million represents less than 1% of the total spending by the district.  Numerators always feel bigger without denominators.

The third claim that inequities in private fund-raising are exacerbating inequities in student achievement pre-supposes that the private giving makes a big difference in the wealthier districts.  It also pre-supposes, contrary to the bulk of rigorous research, that variation in spending is a significant factor in explaining variation in achievement.  It’s not.  So, if private giving is a tiny portion of total spending — even in the wealthy districts — and per pupil spending does not significantly account for achievement, it’s not clear why the article would fret that inequities in giving were a problem for the achievement gap.  But the article does, quoting state Supt. of Public Instruction Jack O’Connell, “Parents in well-to-do communities can raise significant sums of money to augment their local schools’ budgets, while schools in low-income neighborhoods fall further behind. This is part of the reason that we have an achievement gap in California. We have an economic and moral imperative to close this gap.”

The only way the money myth will fade is if reporters and newspapers are held accountable for repeating it. 

5 Comments | journalism, money myth | Tagged: , , , | Permalink
Posted by Jay P. Greene

Priest and Teacher Sex Scandals in Perspective

April 19, 2008

As Pope Benedict XVI visited with victims of sexual abuse by priests in Boston yesterday, the news was again filled with concerns about the widespread and persistent nature of these abuses in the Church.

Indeed, these acts are deplorable and intolerable.  But it might also be useful to put in perspective how widespread sexual misconduct by priests is by comparing the problems in the Church with how widespread sexual misconduct is by teachers in schools. I know this is an unpleasant topic, but a recent analysis by the Associated Press (updated link to AP story) suggests that sexual misconduct among male teachers is at least as common as among male priests.

The AP collected information on the number of teachers who lost their licenses between 2001 and 2005 due to sexual misconduct. They find that 2,570 licenses were revoked among a teaching workforce of almost 3 million. They compare this to the priest abuse scandals where they claim that 4,400 priests were accused of misconduct between 1950 and 2002 out of a priest workforce of 110,000.

At first glance it would seem that teacher scandals are much less common than priest scandals, but the priest numbers are of allegations, not license revocations, and are over a 53 year period instead of a 5 year period. If we adjust for the time difference, there were .76 priest scandal allegations per year per 1,000 priests. For teachers there were .17 teacher license revocations per year per 1,000 teachers. But if we further adjust for the fact that more than 9 in 10 teacher perpetrators are male, while all priest perpetrators are male, there were .77 male teachers who lose their license each year for sexual misconduct for every 1,000 male teachers employed.

The rate of sexual misconduct among male teachers is about as high as among male priests. And given that we are comparing license revocations for teachers to allegations for priests, the rate of misconduct among male teachers may be considerably higher than among male priests. As the AP story notes, relatively few instances of teacher sexual misconduct result in a license revocation.

The investigative reporter, Scott Reeder, has also been doing work in this area and is currently documenting the rate at which teachers who are accused of misconduct actually lose their licenses. You can find his reporting here:

http://www.thehiddencostsoftenure.com/

11 Comments | scandal | Tagged: , , | Permalink
Posted by Jay P. Greene

Proximity and Power

April 19, 2008

If you stand on the steps of a state capitol building and throw a rock (with a really strong arm), the first building you can hit has a good chance of being the headquarters of the state teacher union.  For interest groups, proximity to the capitol is a way of displaying power and influence.  The teacher union, more than any other interest group, strives to be the closest.  They want to remind everyone that among powerful interest groups, they are the most powerful – a prince among princes.

To see who has the most powerful digs, Jonathan Butcher and I actually bothered to measure just how close interest group offices are to state capitol buildings.  We started with a list of the 25 most influential interest groups, as compiled by Fortune magazine.  We then used Google Maps to plot the location of the state offices of those 25 interest groups and measured the distance to the capitol building. 

The results are illuminating.  Of the 25 most influential interest groups, the teacher union is the closest in 14 of the 50 states.  The labor union, AFL-CIO, is the closest in 7 states.  The American Association of Retired Persons (AARP) and Nation Federation of Independent Business are the closest in 5 states, each.  The trial lawyers lobby, the American Association for Justice, is the closest in 4 states.

The teacher union is among the four closest interest groups in 27 states.  The trial lawyers are in the top four in 22 states, followed by the AARP in 20 states and the AFL-CIO in 19 states. 

In Nebraska, the teacher union office is only 210 feet from the capitol building.  In Pennsylvania it is only 312 feet away.  In Alabama, Delaware, and South Dakota, the teacher union headquarters is about 500 feet away. 

If we gave four points for being closest, three for being the second closes, two for being third closest, and one for being the fourth closest, teacher unions would have a total of 85 points.  No other group would have more than 60 points.  Only four of the 25 groups would have above 40 points, with the trial lawyers, AARP, and AFL-CIO joining the teacher union in this elite group.  But even among this expense-account dinner crowd, the teacher union is way ahead.

The teacher union doesn’t strive to be the closest because the extra time it takes to walk or drive a few more blocks will make the difference in a pivotal vote.  They strive to be the closest because it is a visible display of their power and influence.  It’s a symbol of the connections and resources they can devote to something as trivial as having the closest office, just like the status obtained from having the best seats at a concert or sporting event.  If they can blow money on that, do you really want to mess with them?

But much of the power of interest groups is little more than bluff.  It is to their advantage to exaggerate their power and influence precisely because doing so enhances the power and influence they actually have.  Yet despite all of the fear and trembling among politicos of the consequences of crossing the teacher union, they can be beaten. 

In fact, it is striking how often the teacher union loses even with all of its resources and displays of power.  In the face of union opposition to structural reforms, there are now 21 voucher or tax credit programs in 13 states sending students to private schools at public expense.  There are more than one million students attending charter schools in the 40 states that have charter programs.  Merit pay for teachers is being tried in New York, Florida, Texas, Nashville, and several new districts under a federal pilot program. 

The teacher union doesn’t want people to think that they can lose.  They want to impress folks with their prime real estate and well-heeled lobbyists.  But eventually it is hard to sustain really bad ideas in public policy – and the teacher union has embraced some really bad ideas. Eventually the “puffery” of swank offices succumbs to the substantive pursuit of good policy.  In the end, the power of the teacher union may be, in the words of Chairman Mao, little more than a paper tiger – or a well-placed building.

4 Comments | teacher union | Tagged: , , | Permalink
Posted by Jay P. Greene

Next Entries »