What Is “Merit”?

September 16, 2009

(Guest post by Greg Forster)

As JPGB’s friend Marcus Winters notes on NRO today, the Obama administration has been staying the course on this issue, denying “Race to the Top” funds to states that disallow the use of objective measurement in evaluating teachers. Marcus also rightly links this to the topic of merit pay, which the president repeatedly embraced during the campaign. One of the biggest obstacles to enacting real merit pay is making sure that the measurement of “merit” really measures merit. Too many experiments with “merit pay” have really been experiments in peer evaluation pay, grade inflation pay, and so on. Check out Marcus’s article for more.

Tampa Tribune Op-Ed

September 3, 2009

Marcus Winters and I have an op-ed in this morning’s Tampa Tribune on how Florida’s McKay voucher program for special education students has restrained the spiraling growth in special education enrollments in public schools.  We write:

In Florida, as in most other states, schools receive additional funding for each student identified as disabled. Often, these additional resources are greater than the actual cost of providing special-education services, giving schools a financial incentive to increase their diagnoses.

The financial incentive to misdiagnose is particularly apparent when classifying students as having a specific learning disability (SLD). That’s because SLD is the most common, the most ambiguous, and the least costly category of special education. In many cases, school officials might simply be trying to get extra resources to help struggling students. But the net effect is the misclassification of a huge number of students as having an SLD.

The McKay program reduces the financial incentive for Florida’s schools to misdiagnose learning disabilities by placing revenue at risk whenever a student is placed into special education…

In our new study, we found as the number of nearby, McKay-accepting private schools increases, the probability that a public school will identify a student as having an SLD decreases significantly. The program reduced the probability that a fourth-, fifth-, or sixth-grader in a school facing the average number of nearby private options was diagnosed as SLD by about 15 percent.

Special Ed Vouchers Restrain Growth in Disabilities

August 18, 2009

Marcus Winters and I have a super-awesome study released today by the Manhattan Institute.  It shows that offering disabled students special education vouchers reduces the likelihood that public schools will identify students as disabled.

This isn’t what Andy Rotherham and Sara Mead expected.  They claimed in a 2003 report for the Progress Policy Institute that: “special education vouchers may actually exacerbate the over-identification problem by creating a new incentive for parents to have children diagnosed with a disability in order to obtain a voucher.”

It didn’t. The reason special education vouchers restrained growth in disabilities, rather than exacerbate it, is that the vouchers check public schools’ financial incentives to identify more students as disabled.  Public schools may get additional subsidies when they shift more students into special education, but if they then make students eligible for special education vouchers, they risk having those students walk out the door with all of their funding.  It makes the public schools think twice before over-identifying disabilities for financial reasons.

And outside of the DC bubble, schools control the process of whether students are identified as disabled — not parents.  So, if we can check the positive financial incentives that public schools have for over-identifying disabilities, we can significantly slow growth in special education.

Nearly 1 in 7 students nationwide is now classified as having a disability.  That’s 63% more than three decades ago.  It’s clear that this huge increase in disabilities was not caused by a true increase in the incidence of disabilities in the population.  No plague has afflicted our children over the last three decades to disable two-thirds more of them.

Instead, non-medical factors have been driving special education enrollments higher.  Chief among these is the financial incentives we offer schools in most states to shift more students into special education by providing additional subsidies for each student classified as disabled.

Some states have reformed their special education funding formulas to end these financial rewards for higher special education rolls.  Greg and I reported in a 2002 study that states that continued to pay schools per student identified as disabled had much higher rates of growth in special education than states that had reformed their funding formulas.  Elizabeth Dhuey of the University of Toronto and Stephen Lipscomb of the Public Policy Institute of California have confirmed these findings.

Julie Cullen of UC San Diego has found that “fiscal incentives can explain over 35 percent of the recent growth in student disability rates in Texas.”  And Sally Kwak, a student of David Card at UC Berkeley and now a professor at U of Hawaii, finds a significant slow-down in special education enrollments when California reformed its funding system.

The new study Marcus and I released today builds upon this growing research by showing yet again that public schools strongly consider non-medical factors when deciding whether to classify students as disabled.  I don’t mean to suggest that all school officials are conscious of these incentives or acting with evil intention.  But it is clear that the system in which they operate and their actions are shaped by these financial incentives.

If we discovered that hospitals were filling their beds with healthy people who just felt a little tired in order to obtain additional government subsidies, we would be outraged and demand dramatic reforms.  Public schools are doing the same and it is time we get outraged and demand reforms.

Brummett About to Throw Another Tantrum

July 15, 2009

John Brummett, a columnist in a local paper in Northwest Arkansas called the Morning News, posted on his blog that he is going to write another column attacking me.  At least he gives fair warning.

In 2007 he wrote an angry column in response to a report I co-wrote with Marcus Winters about teacher pay.  The column concluded:

What’s inherently nonsensical — no, breathtakingly offensive — is for someone interested in those very reforms to be so politically lead-footed as to write an article saying teachers are paid plenty already, and do so while he pulls down $160,000 or more in a public education faculty position himself, and while he is underwritten by a foundation created by wealthy heirs of a fortune gleaned in part from low employee wages and sparse employee benefits.

At the time I hadn’t started this blog, so I didn’t think there was a reasonable forum to address his piece.  But now that the blog is pulling in a daily readership that is not too far off the daily readership of his column in the Northwest Arkansas Morning News (daily circulation 33,582), I’ll respond to the old column and anticipate his new one.

Other than being angry himself and asserting that I had said “something crazy that makes every school teacher in Arkansas throw an eraser across the classroom,” it is not clear what substantive objection Brummett has to what I wrote.  He never disputed the accuracy of the facts I presented on teacher pay, nor could he.  The numbers I presented were taken directly from the U.S. government’s Bureau of Labor Statistics (BLS). 

I suppose one could object to how the BLS calculates hourly pay, based on the argument that it fails to fully capture teacher hours worked outside of school more than it fails to capture hours worked outside of the office by other professionals and white collar workers.  But we addressed that concern in the report by comparing teacher wages to those of other white collar and professional workers on a weekly basis.  Teachers still earn more than other white collar and professional workers.  Teachers do earn less on an annual basis, as we said, but having breaks during the Winter, Spring, and Summer is worth money.  If you don’t think it is, how do you think teachers would feel if we asked them to work all year for the same annual pay they get now?

In addition, I never said that teachers are overpaid, despite Brummett’s suggestion to the contrary by describing my view incorrectly as “teachers get paid plenty already.”  In fact, in the report we explicitly stated: “we offer no opinions on the proper level of pay for public school teachers. We are simply offering facts, almost entirely obtained from an agency of the federal government, that we believe ought to be included in any policy discussion about teacher pay.”  Instead, our point, other than providing descriptive information, was to suggest that teacher pay was roughly comparable on an hourly or weekly basis to that of other white collar and professional workers. 

We did provide an exploratory regression analysis showing no relationship between the level of teacher pay and student outcomes, controlling for observed demographics.  And we did suggest that higher pay might yield better student achievement if it were more explicitly connected to achievement via a merit pay system.  But these arguments do not suggest that teachers are paid too much, only that we should explore paying them differently.

What’s even stranger about Brummett (and others) being offended by my report, is that it is not clear what would be bad about saying that teachers are reasonably well-compensated.  In business schools they routinely brag about how well-paid their graduates are.  Doing so helps them attract more and higher quality applicants.  Why wouldn’t we want to do the same in Education colleges?  I understand that some teachers and their unions may nurse the false grievance of being paid significantly less than other professionals in order to gain leverage in seeking pay increases in the future.  But why should researchers, journalists, and Education college officials suppress accurate and truthful information to assist them in that effort?

Brummett may get angry again tomorrow.  He may throw his column across the room.  He may talk about how much I get paid.  He may offer more political advice, as if researchers should tailor their reporting of the facts to suit political interests.  He may say I’m controlled by the Waltons or Keyser Soze

But he can’t change the facts.  The BLS numbers are what they are.  He can try to distract his readers from that evidence, but he can’t make a substantive argument against what I’ve reported.

UPDATE —Sure enough, Brummett threw his tantrum.  He opens with name-calling: “He’s right-wing and quite the zealous advocate of many education reform notions.”  

Then he assigns to me responsibility for all sorts of things that aren’t actually attributable to me.  For example, he says (dripping with sarcasm): “He gives [teachers] summers off and calculates their hours of actual classroom instruction and concludes that he knows people in other professional fields who aren’t doing as well or significantly better.” 

I didn’t do any of those things.  Teacher contracts with schools give them the summers off.  The U.S. Bureau of Labor Statistics (BLS) calculates their hourly and weekly pay.  The BLS reports that teachers, on average, make more than other white collar and professional workers on both an hourly and weekly basis.  I just repeated what the BLS reported.

He continues falsely attributing to me claims that were not invented by me: “He faulted [Fayetteville schools] for spending federal stimulus dollars not to stimulate the economy, but to pay teachers what he assumes to be twice their usual hourly rate for something they would have been doing anyway, and for much less, without the stimulus.”  (emphasis added)  I didn’t assume that teacher pay was doubled with the stimulus dollars.  The Northwest Arkansas Times reported that fact and I, again, just repeated it.

Finally, he makes the case for this use of stimulus dollars: “This is a new and different program that wouldn’t have been undertaken without the extra Title 1 money from the stimulus, [district officials] say. This will be high-intensity summer session with innovative techniques and individualized instruction and counseling, they say.”

I never disputed that the program might be a beneficial one.  As I wrote in my initial post on this topic: “The Leap Ahead program may well be a good one.”  My objection is to paying teachers twice their normal rate (as reported by the NWAT) and three times what teachers in neighboring Springdale are being paid for the same program.  Nothing in Brummett’s column justifies that.  And he conveniently neglects to mention how Springdale teachers are being paid 1/3 as much for the same thing.

It’s clear that John Brummett uses his column to prosecute his own personal, political agenda.  That’s acceptable for a columnist, but normally they have to be constrained by facts and logic in doing so.  He can’t falsely attribute to me claims that are not my own.  And he can’t switch the issue from doubling (or tripling) teacher pay for a program to the desirability of that program.  At least, his newspaper shouldn’t let him do these things with their paper. 

Who exactly is the zealot here — the person repeating the factual claims of the BLS and the Northwest Arkansas Times or the person omitting crucial facts, falsely attributing claims, and changing the subject?

(Edited for typos.  See a follow-up post here.)

Marcus Winters on School Choice Savings

January 21, 2009


(Guest post by Greg Forster)

Since we’re on a money kick this week, let’s combine that theme with education (this blog is still about education, right?) and note that our friend Marcus Winters has an article on NRO today on how vouchers save money.

For those looking to dig deeper, here’s an analysis of the fiscal impact of every school choice program from 1990 through 2006. Every program was at least fiscally neutral, and most saved money.

The Growing Charter School Consensus

January 15, 2009

A string of high quality studies is finding that students benefit academically from attending a charter school rather than a traditional public school. 

First we had a random-assignment study of Chicago charter schools by Caroline Hoxby and Jonah Rockoff that found “that students in charter schools outperformed a comparable group of lotteried-out students who remained in regular Chicago public schools by 5 to 6 percentile points in math and about 5 percentile points in reading.” 

Then Hoxby conducted a random-assignment study of charter schools in New York City and found: “that the average effect of the charter schools on math is 0.09 standard deviations for every year that a student spends in his or her charter school. The average effect on reading is 0.04 standard deviations for every year that a student spends in his or her charter school.” 

Then Kevin Booker (Mathematica), Tim Sass (Florida State), Brian Gill (Mathematica), and Ron Zimmer (Rand)used a well-designed instrumental variable analysis to see whether charter middle-schoolers who continue to charter high schools are more likely to graduate.  They are. 

And most recently a random-assignment analysis of charter schools in Massachusetts led by Tom Kane at Harvard and Josh Angrist at MIT found that charter school students accepted by lotteries significantly outperformed their counterparts in traditional public schools, unless the charter school was operated by the teacher unions.

In light of these high quality studies, it is harder to oppose charter schools on a scholarly basis.  And with the clear support of charters from the incoming Obama administration, it is getting harder to opposed charter schools on a political basis — at least at the national level.

But don’t expect to see the teacher unions waving a white flag despite their losses in research and national politics.  They don’t need facts or the support of the US Department of Education so long as they continue to dominate local school politics. 

And that is exactly why they have focused on organizing local charter schools to neutralize the threat to their grip on local school politics.  As my colleague Marcus Winters writes today in the New York Post, the unions managed to organize two successful charter schools in New York City.  The fact that union-run charter schools in Massachusetts trailed the non-union charters in performance is not of concern to the unions.  It isn’t about student achievement; it’s about keeping their hold on power even as the facts pile up against them.

The Irony of Social Promotion

January 9, 2009

In the current issue of the Economics of Education Review, Marcus Winters and I have an article about the use of exemptions to Florida’s test-based promotion policy.  Under Florida’s policy students need to perform above a certain level on the 3rd grade reading test to automatically be promoted to 4th grade.  If  students score below that level they can still be promoted if they are granted one of various exemptions.  Some of those exemptions are objectively measured, like scoring well on an alternative test or having certain special ed or English Language Learning classifications.  But other exemptions are more subjectively determined, like having a portfolio of work worthy of being promoted.

Marcus and I looked at who received those exemptions and whether being exempted was beneficial.  We found that African-American and Hispanic students were less likely to receive exemptions and get promoted, controlling for other factors.  That is, minority students with the same test scores and economic status were less likely to be exempted from retention if they fell below the testing threshold.  The test-based policy is not racially biased, since all students who lack the academic skills to pass the test may be retained.  The bias is introduced in who gets exempted from that test-based policy.

And the irony of it all is that failing to receive an exemption actually benefited those minority students academically.  That is, students who were denied the exemption and repeated third grade outperformed their promoted colleagues on achievement tests two years later.  The retained students had more academic skill at the end of 4th grade than their comparable promoted peers at the end of 5th grade — despite being exposed to one less grade of curriculum. 

Minority students denied the exemptions may have been the vicitms of discrimination, but they ended-up making greater academic progress as a result.  Receiving those exemptions wasn’t doing many of the other students any favors.

The St. Pete Times has an article on the study today and had a blog post recently.

Grading New York

November 13, 2008

(Guest post by Greg Forster)

Our old friend and colleague Marcus Winters has just released a study on New York City’s school grading program:

In 2006-07, New York City, the largest school district in the United States, decided it would follow several other school systems in adopting a progress report program. Under its program, the city grades schools from A to F according to an accumulating point system based on the weighted average of measurements of school environment, students’ performance, and students’ academic progress.

The implementation of these progress reports has not been without controversy. While many argue that they inform parents about public school quality and encourage schools to improve, others contend that grades lower morale at low-performing schools. To date there has been too little empirical information about the program’s effectiveness to settle these questions.

Schools that recieve D and F grades repeatedly are subject to takeover by the city. A previous study (Rockoff and Turner 2008) found positive results from the program but lacked student-level data. Marcus’s study has got student-level data, regression discontinuity – the whole smash. Tale of the tape:

Students in schools earning an F grade made overall improvements in math the following year, though these improvements occurred primarily among fifth-graders.

Students in F-graded schools did no better or worse in English than students in schools that were not graded F.

Whatever problems NCLB may have, school accountability does work in places where state and local government have the political will to do it seriously. Even in places where the problems seem intractible, like New York City.

EMTs are standing by in case certain people’s heads explode.

A Few Comments

September 9, 2008

It must be the back to school season because there are a lot of interesting education pieces on the web.  I thought I’d just mention and briefly comment on some:

  • On Matt Ladner’s Little Ramona’s Gone Hillbilly Nuts about Diane Ravtich’s new-found enthusiasm for teacher unions and hostility to charter schools and merit pay — I posted this comment on his piece: “I liked Left Back, Language Police, and much of her historical work. That’s why it’s so disappointing to read what she is writing these days. From her earlier work one would never have guessed that she would accuse people who favor merit pay, reduction in teacher tenure rights, and charter schools of plotting to destroy public education.  And for someone whose past work relied on rigorous scholarship, it is shocking to see these new claims made without any evidence that merit pay, weaker tenure, and charter schools harm public education, let alone destroy it.  Other than the fact that Bloomberg and Klein support these policies, it is not clear why Diane Ravitch opposes them.”
  • Marcus Winters has a great piece on National Review Online about how reforming the teacher compensation system is the key to improving teacher quality and, in turn, student achievement.
  • Thomas Hibbs has a not-so-great piece on National review Online about how “the true teacher cannot simply be an instrument of the wishes of the student’s family.”  He’s right that parents can sometimes try to shield their children from burdens by lowering academic expectations and that teachers need to strive for excellence regardless.  But it’s unrealistic to expect that we can build an educational system based on “the teacher’s love.”  Parents, whatever their shortcomings, are more likely to be effective advocates for a child’s progress than even well-intentioned and well-trained teachers because the parents have a love for children that we cannot realistically expect from teachers. 
  • I don’t have time to comment on them, but you should also check out the rest of the National Review Online pieces, including those by Checker Finn, Neal McCluskey, Mike Petrilli and Amber Winkler, and Susan Konig.

Eduwonkette and Eduwonk Aren’t Edumarried?

July 8, 2008

The New York Sun had a nice profile yesterday of Eduwonkette.  Well, it’s not exactly a profile because Eduwonkette writes anonymously.  In the article some folks complain that her anonymity is a problem: “A co-director of the Education Sector think tank, Andrew Rotherham, suggested on his blog Eduwonk that Eduwonkette might be unfairly pretending to be unbiased because she has ‘skin in the game… It’s this issue of you got all this information to readers, without a vital piece of information for them to put it in context.'”

I think Andy’s mistaken on this. (Did they have some kind of edu-break-up?)  The issue is not who Eduonkette is, but whether she is right or not.  Knowing who she is does not make her evidence or arguments any more or less compelling.  I wish we all spent a whole lot less time analyzing people’s motives and a whole lot more time on their evidence and arguments. 

The only major problem with anonymity is lack of responsibility for being wrong.  There is a reputational price for making bad arguments or getting the evidence wrong that Eduwonkette avoids paying professionally — although she does pay a reputational price to the name brand of Eduwonkette.

Speaking of being wrong, Eduwonkette knocks the study Marcus Winters, Julie Trivitt, and I released today through the Manhattan Institute.  She complains: “It may be an elegantly executed study, or it may be a terrible study. The trouble is that based on the embargoed version released to the press, on which many a news article will appear today, it’s impossible to tell. There is a technical appendix, but that wasn’t provided up front to the press with the glossy embargoed study. Though the embargo has been lifted now and the report is publicly available, the technical appendix is not.”

This isn’t correct.  Embargoed copies of the study were provided to reporters upon their request.  If they requested the technical report, they could get that.  Both were available well in advance to reporters so that they could take time to read it and circulate it to other experts before writing a story.  Both the study and the technical report were made publicly available today (although there seems to be a glitch with the link to the technical report that should be fixed within hours).  The technical report can be found here.

And while we are on the subject of Eduwonkette being wrong, her attacks on test-based promotion policies are overdone.  The Jacob and Lefgren paper does raise concerns, but there is more positive evidence from the experience in Florida.  As I wrote in a previous post: “In a study I did with Marcus Winters that was published in Education Finance and Policy, we found that retained students significantly outperformed their comparable peers over the next two years.  In another study we published in the Economics of Education Review, we found that schools were not effective at identifying which students should be exempted from this test-based promotion policy and appeared to discriminate in applying these exemptions.  That is, white students were more likely to be exempted by school officials in Florida from being retained, but those students suffered academically by being exempted.”

Our results may actually be consistent with what Jacob and Lefgren find.  We find academic benefits for students retained in third grade.  They find: “that grade retention leads to a modest increase in the probability of dropping out for older students, but has no significant effect on younger students.”  It could be that test-based promotion is more beneficial when done with younger students.  It could also be that the policy has positive effects on achievement with some cost to graduation. 

And particularly severe problems with the integrity of test results used for promotion decisions in Chicago may limit the ability to generalize from Chicago’s experience.  In Chicago it may have been easier to move retained students forward by cheating on the next test than actually teaching them the basic skills they need to succeed in the next grade.

Besides, I’m sure that Edwuonkette wouldn’t put too much stock in Jacob and Lefgren’s non-peer-reviewed paper released straight to the public.  I’m sure she would be consistent in her view that: “By the time the study’s main findings already have been widely disseminated, some sucker with expertise in regression discontinuity may find a mistake while combing through that appendix, one that could alter the results of the study. But the news cycle will have moved on by then. Good luck interesting a reporter in that story… So as much as I like to kvetch about peer review and the pain and suffering it inflicts, it makes educational research better. It catches many problems and errors before studies go prime time, even if it doesn’t always work perfectly.”  

Or do these standards only apply to studies whose findings she doesn’t like?   If Eduwonkette isn’t careful she might get a reputation.

%d bloggers like this: