Jay P. Greene's Blog

The AFT’s Leo Casey of union cue-card check and sock-puppet fame has written a blog post for his “steakholders” once again accusing me of cherry-picking.  The last time Leo accused me of cherry-picking voucher studies I produced what I believe are comprehensive lists of random-assignment voucher participant and high-quality voucher competitive effect studies.  Given his inability to substantiate that cherry-picking charge, I’m a bit surprised to see that he is a glutton for punishment and wants to make the charge again.  I guess Leo has joined with Shari Lewis, Lambchop, and all his sock-puppet friends to make this cherry-picking charge another song that never ends.

This time the issue is whether teacher unions tend to raise costs and lower student achievenement.  Leo noticed my guest posts over at Flypaper on this and asserts: “if you think that the scholarly literature on the subject is a guide, it clearly comes down in a place quite different from that suggested by Greene.”

Leo’s claim hinges entirely on what “scholarly literature” means and whether all “studies” should be treated equally.  For example, I could claim that the scholarly literature shows that candy improves student achievement, citing as scholarly literature papers written by my 5th grade son and friends whose research design involved describing how smart they felt after eating candy.

Leo doesn’t go quite that far but he does cite a study by the AFT’s very own Howard Nelson that makes a cross-sectional comparison of test scores controlling for a handful of observed demographics.  He also cites literature reviews that consist mostly of these cross-sectional analyses controlling for observed demographics. 

The problem is that there is a serious design flaw with these studies — unobserved factors that are associated with unionization may also be associated with student achievement.  For example, wealthier communities may be more likely to produce unionization because those communities have the wealth to bear the higher costs associated with unionized teachers.  Wealth may be associated with higher student achievement, but our controls for wealth (free lunch status) may not fully or accurately capture the differences in wealth.  So, unobserved and uncontrolled factors would bias the results from these cross-sectional studies.

Caroline Hoxby’s study, upon which I base my claims, employs a vastly superior research design that addresses this problem.  I’ll let her describe the problem and how she solves it:

“The … most serious obstacle is the identification problem caused by the difficulty of differentiating between the effects of a union on a school and the characteristics of a school  that make a union more likely to exist.  Even after controlling for observable characteristics of a school district such as demographics, there are presumably unobservable school characteristics associated with unionization.  The unobservable school characteristics that promote unionization may themselves affect the education production function….  My third, and probably best, attempt to solve the identification problem combines differences-in-differences and instrumental variables estimation.”

Her instrumental variable strategy involves using changes in state laws regarding unionization to derive unbiased estimates of when schools would unionize.  The change in the state law would help predict whether a school unionizes without being associated with the academic achievement in that school.  This is a far better way to estimate the effect of unionization than simply looking at whether unionized schools have higher or lower scores, since the scores and other factors associated with school quality could themselves be causing the unionization.

I’d put much more confidence in this rigorously designed study than a dozen weakly designed cross-sectional analyses.

But even if Leo insisted upon relying on the literature reviews he cites rather than the higher quality research, he would have to accept some results that aren’t very flattering to teacher unions.  Those lit reviews find that unionization raises the cost of education by about 8% to 15%.  In addition, they find that unionization tends to hurt the academic achievement of high-achieving and low-achieving students while benefiting more typical students found in the middle of the ability distribution. 

As Leo’s authority, Eberts, Hollenbeck and Stone, put it: “While on average students fare at least as well, if not better, in unionized schools, atypical students – students well below or above average ability – do appear to fare less well because instructional settings are more standardized, less individualized in unionized schools.”

So, if Leo wants to say that unions exacerbate the achievement gap for disadvantaged minority students while driving up costs, I guess he can rely on that literature review.  I prefer to rely on Caroline Hoxby’s rigorously designed study in a top economics journal.

This entry was posted on Wednesday, May 20th, 2009 at 7:19 pm and is filed under teacher union. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.