Violating the Denominator Law

December 2, 2008

Sean Corcoran, who is guest blogging for the blogger formerly known as Eduwonkette, may have to go to education research jail because he violated the Denominator Law today.  For those of you unfamiliar with the Denominator Law from my previous post on the topic (and ignorance of the law is no excuse) it is: “No one should be allowed to highlight numerators without also presenting denominators.  That is, it is often misleading to describe a big number without putting that number in perspective.”

So, Sean is all worried about private donations to public schools creating or exacerbating inequities in funding.  He references a report about California (and it had better be peer-reviewed or the blogger formerly known as Eduwonkette will throw a fit) that finds: “contributions to California school foundations rose from $123 million in 1992 to $238 million in 2001.”  He does helpfully add that $238 million only amounts to $40 per pupil.  But he doesn’t fully comply with the Denominator Law because he fails to point out that $238 million only represents .4% of the $52.2 billion in total public school revenue in California in 2001.

It’s not the average amount of private giving in California that really worries him.  What concerns him is that these donations are concentrated in wealthy areas: “Of course—as Brunner and Imazeki point out—these contributions are far from evenly distributed. Donations are strongly related to family income, and in some cases they are quite high, at more than $250 to $500 per student. (You can read about the $3.3 million education foundation in Santa Monica-Malibu Unified School District here).”

Here, your honor, is where he flagrantly breaks the Denominator Law.  He suggests that $250 to $300 per pupil, as illustrated by $3.3 million in private giving to public schools in Santa Monica, is “quite high.”  Without a denominator, it’s hard to judge how high $3.3 million in Santa Monica really is. 

Let me help.  According to the School Matters web site operated by Standard and Poor’s, Santa Monica has 12,191 students.  The private contributions Corcoran mentions amount to $271 per pupil — within his $250 to $300 range.  But total revenue for Santa Monica public schools amounted to $11,062 per pupil as of 2006.  Private contributions of $271 amount to only 2.4% of total revenue — not exactly “quite high.”

And this private giving hardly accounts for resource differences between Santa Monica and the average district in California.  According to School Matters the average district in CA had total revenue of $9,553 as of 2006, $1,509 less than in Santa Monica.  If Santa Monica received $271 in private donations compared to $40 for the average California district, the extra $231 could only account for about 15% of the extra resources Santa Monica possesses. 

If this is the worst case that folks can muster, it hardly seems like private giving is a significant contributor to resource inequities.  We only gain this appropriate perspective when we comply with the Denominator Law — so be sure to follow the law out there.

Bloggers Shouldn’t Have Rapper Names

October 8, 2008

In my last post I described Jennifer Jennings as the blogger formerly known as Eduwonkette.  I had thought we only had to call her Eduwonkette when we didn’t know who she was.  But I guess she continues to go by her rapper name, Eduwonkette.  And Aaron Pallas, an otherwise respectable scholar, continues to call himself Skoolboy — with a k!  And I guess they are both cribbing (in the non-rapper meaning) from Eduwonk, who we’ve always known to be Andy Rotherham.

I find the use of rapper names by bloggers to be downright silly.  It’s especially silly when accompanied by self-aggrandizing cartoons and graphics.  Here at Jay P. Greene’s Blog we’ve gone for a minimalist approach, both out of laziness and an aesthetic vision that tried to put the focus on content.

But if a bunch of other folks are going to continue to call themselves rapper names and have cartoon graphics to represent themselves, maybe I should do the same.  Perhaps I should go by my rapper name — DJ Super-Awesome.  And maybe we should use Thundarr the Barbarian graphics to represent ourselves.  I call the image of Ookla and I’ll let Greg and Matt fight over who gets to be Ariel.

Policymaking By Anecdote

October 7, 2008

Is it good policy to reduce barriers to firing sub-par teachers?

According to Jennifer Jennings, the blogger formerly known as Eduwonkette, the answer is no.  We need to preserve teacher tenure, she argues, because she has found an example of a really great teacher, Art Siebens, who was fired when his DC school was reconstituted.  His case is “haunting for the glimpse it offers into the brave new world of unchecked principal autonomy.”

Well, Siebens wasn’t actually fired.  He wasn’t re-hired at the same school and was instead offered a job teaching a different science course at a different DC public school.  Don’t fret ye of weak hearts — continued employment for teachers is still essentially guaranteed even if not in the school and class of their choosing.

It’s puzzling why Siebens wasn’t re-hired given that he was an award winning teacher with what appears to be a strong record of excellent work.  But the fact that he wasn’t is hardly evidence against DC superintendent Michelle Rhee’s proposal to offer teachers significant pay increases if they give up tenure.  Perhaps there is more to Siebens’ story than is publicly known.  

More importantly, the case of Art Siebens is not evidence against abandoning tenure because it is a single case.  The plural of anecdote is not data.  We shouldn’t make policy by referencing anecdotes.  Instead, we should look “through the lens of social science,” as a wise person once wrote, and consider systematic evidence when formulating education policy. 

The remarkable investigative reporting by Scott Reeder has powerfully documented the problems with teacher tenure.  After filing 1,500 Illinois Freedom of Information Act requests with the state board and all 876 Illinois school districts, Reeder uncovered the following:

1) “Of an estimated 95,500 tenured educators now employed in the state [of Illinois] an average of only seven have their dismissals approved each year by a state hearing officer. Of those seven, only two on average are fired for poor job performance. The remainder is dismissed for issues of misconduct.”

2) “Of Illinois’ 876 school districts only 61, or 7 percent, have ever attempted to fire a tenured faculty member since the teacher evaluation reforms were imposed 18 years ago.”

3) “Of those 61 school districts, only 38 were successful in actually firing a teacher.”

4) “Not only is it exceedingly rare to fire a tenured teacher in Illinois, but it also is extraordinarily expensive. In fact, Illinois school districts that have hired outside lawyers in these cases have spent an average of more than $219,000 in legal fees during the last five years.”

5) “In the last 10 years, about 477,000 evaluations of Illinois tenured teachers have been performed, but only 513 received unsatisfactory evaluations… In other words, only 1 out 930 evaluations result in a tenured teacher receiving an ‘unsatisfactory’ rating.” Conducting those evaluations consumed 2.5 million administrative hours.

OK, so it is next to impossible to fire tenured teachers.  And we also know from systematic evidence that the quality of the teacher is the single most important factor within school control to influence student academic improvement. (See for example the research referenced here.)

Unless we believed that all but .007% of tenured teachers are doing a solid job, the current system is clearly keeping incompetent teachers in the classroom.  Any meaningful reform strategy has to involve getting rid of dud teachers and attracting better teachers as replacements.  Rhee’s proposal to increase pay in exchange for greater flexibility in terminating sub-par teachers seems like a promising idea to do just that.  The higher pay might attract better new people into teaching and the flexibility on termination could remove bad teachers from the classroom. 

Of course, the blogger formerly known as Eduwonkette makes a fair point when she asks, “Why are you confident that principals will always – or even often – pick the ‘best teachers?'”  For a system with reduced tenure protections to work, principals would also have to be properly motivated to distinguish between effective and ineffective teachers.  But this could be done either through meaningful merit evaluation and rewards for principals or through market accountability in choice programs.  If continued employment or pay raises for principals depended upon identifying effective and ineffective teachers, they are unlikely to let talented teachers like Art Siebens go and are likely to get rid of duds.

But we should all accept that any system of hiring and firing teachers will have its injustices.  The status quo tenure system has the injustice of protecting bad teachers in their jobs.  And if cuts have to be made it is newer teachers who have to be let go, even if they are better teachers than their senior colleagues.  A system like Rhee proposes will occasionally mistakenly let go of a good teacher.  But with well-designed incentives for the principals this should be the exception and not the rule.

Besides, we have to ask ourselves:  how many kids do you want to condemn to an ineffective teacher to avoid the possibility of unjustly terminating a good teacher?  If we care more about the kids than the adults in schools, then the injustice to the students should matter much more to us than the possible injustice to a few teachers.

Eduwonkette or Jennifer Jennings, Makes No Difference

August 26, 2008

I’m glad that Eduwonkette decided to end her anonymity and identify herself as Jennifer Jennings, a sociology graduate student at Columbia University.  I’m not glad because I think it was inherently wrong for her to blog anonymously.  As I’ve previously written: “The issue is not who Eduonkette is, but whether she is right or not.  Knowing who she is does not make her evidence or arguments any more or less compelling.” 

The problem was that Eduwonkette did not share my belief in the principle that we should focus on the veracity rather than the source of claims.  She repeatedly emphasized the credibility of the source of information.  Emphasizing the credibility of sources while blogging anonymously, preventing analysis of her own credibility, was logically untenable and had to end.  I wished that she would end this inconsistency by embracing the view that ideas are true or false independent of their sources.  Instead she has resolved her inconsistency by ending her anonymity.

Now that we know that Eduwonkette is Jennifer Jennings we can see another prominent example of the logical inconsistency of blogging anonymously while focusing on the credibility of sources.  In 2005 Jennings published an article in the American Educational Research Journal that argued that accountability systems encouraged schools to focus on the achievement of “bubble” students — those close to an achievement cutoff — at the expense of high and low achieving students.  She arrived at this conclusion after visiting a school in Texas and observing it for a period of time.  She was aided in drawing this conclusion by using jargon like “neoinstituionalism” and “normative isomorphism,” but I kind of zoned out during that part of the article.  I’m guessing that neoinstitutionalism is bad following my theory that anything starting with “neo” is supposed to be bad while anything starting with “post” is supposed to be good.

A few years later Matthew Springer published articles in Education Next and in the Economics of Education Review that empirically examine Jennings’ claim of “educational triage.”  The Education Next piece actually began with a lengthy quote from Jennings’ article as a foil for its findings that NCLB accountability improved the achievement of “bubble” students, but not at the expense of lower and higher achieving students.

Jennings then took-on her critic, Springer, but she did so as the anonymous blogger Eduwonkette, never revealing that she was attacking the person who criticized her own research.  And her first argument against Springer was that his work was published in Education Next and “Education Next is not a scholarly journal.”  Jennings targets the source of the critique of her own work while concealing her identity to prevent analysis of her as a source.  The irony is too rich.

Why Jennings did not just focus on the version of the paper published in Economics of Education Review or the unabridged version linked to on Education Next, , which would have been free of the unscholarly taint she perceives in Education Next,  is unclear.  It was obviously important for her to discredit Springer’s argument against her own study by attacking the credibility of Education Next as the source of Springer’s argument — all the while preventing assessment of her credibility by doing all of this anonymously.

Let me be clear that I have no problem with Jennings defending her own work anonymously.  Her arguments against Springer are true or false regardless of who she says or doesn’t say she is.  My point is that by arguing her own case anonymously, Jennings betrays the principles that she appears to endorse.  Namely, if the source of information is important in assessing claims, it would clearly be inappropriate to attack your critic without revealing who you are. 

Even now that Jennings has revealed her identity, I hope that she abandons her reliance on assessing the source of claims.  Doing so would justify her past actions and help us move forward in analyzing ideas rather than analyzing people and motives.

Voucher Effects on Participants

August 21, 2008

(This is an update of a post I originally wrote on August 21.  I’ve included the new DC voucher findings.)

Here is what I believe is a complete (no cherry-picking) list of analyses taking advantage of random-assignment experiments of the effect of vouchers on participants.  As I’ve previously written, 9 of the 10 analyses show significant, positive effects for at least some subgroups of students.

All of them have been published in peer reviewed journals or were subject to outside peer review by the federal government.

Four of the 10 studies are independent replications of earlier analyses.  Cowen replicates Greene, 2001.  Rouse replicates Greene, Peterson, and Du.  Barnard, et al replicate Peterson and Howell.  And Krueger and Zhu also replicate Peterson and Howell.  All of these independent replications (except for Krueger and Zhu) confirm the basic findings of the original analyses by also finding positive effects.

Anyone interested in a more complete discussion of these 10 analyses and why it is important to focus on the random-assignment studies, should read Patrick Wolf’s article in the BYU Law Review that has been reproduced here.

I’m eager to hear how Leo Casey and Eduwonkette, who’ve accused me of cherry-picking the evidence, respond.

  • These 6 studies conclude that all groups of student participants experienced reading or math achievement gains and/or increased likelihood of graduating from high school as a result of vouchers:

Cowen, Joshua M.  2008. “School Choice as a Latent Variable: Estimating the ‘Complier Average Causal Effect’ of Vouchers in Charlotte.” Policy Studies Journal 36 (2).

Greene, Jay P. 2001. “Vouchers in Charlotte,” Education Matters 1 (2):55-60.

Greene, Jay P., Paul E. Peterson, and Jiangtao Du. 1999. “Effectiveness of School Choice: The Milwaukee Experiment.” Education and Urban Society, 31, January, pp. 190-213.

Howell, William G., Patrick J. Wolf, David E. Campbell, and Paul E. Peterson. 2002. “School Vouchers and Academic Performance:  Results from Three Randomized Field Trials.” Journal of Policy Analysis and Management, 21, April, pp. 191-217. (Washington, DC: Gains for all participants, almost all were African Americans)

Rouse, Cecilia E. 1998. “Private School Vouchers and Student Achievement: An Evaluation of the Milwaukee Parental Choice Program,” The Quarterly Journal of Economics, 113(2): 553-602.

Wolf, Patrick, Babette Gutmann, Michael Puma, Brian Kisida, Lou Rizzo, Nada Eissa, and Marsha Silverberg. March 2009.  Evaluation of the DC Opportunity Scholarship Program: Impacts After Three Years. U.S. Department of Education, Institute of Education Sciences. Washington, DC: U.S. Government Printing Office. (In the fourth year report the sample size shrunk so that the positive achievement effect barely missed meeting a strict threshold for statistical significance — p < .06 just missing the bar of p < .05.  But this new report was able for the first time to measure the effect of vouchers on the likelihood that students would graduate high school.  As it turns out, vouchers significantly boosted high school graduation rates.  As Paul Peterson points out, this suggests that vouchers boosted both achievement and graduation rates in the 4th year.  Read the 4th year evaluation here.)

  • These 3 studies conclude that at least one important sub-group of student participants experienced achievement gains from the voucher and no subgroup of students was harmed:

Barnard, John, Constantine E. Frangakis, Jennifer L. Hill, and Donald B. Rubin. 2003. “Principal Stratification Approach to Broken Randomized Experiments: A Case Study of School Choice Vouchers in New York City,” Journal of the American Statistical Association 98 (462):299–323. (Gains for African Americans)

Howell, William G., Patrick J. Wolf, David E. Campbell, and Paul E. Peterson. 2002. “School Vouchers and Academic Performance:  Results from Three Randomized Field Trials.” Journal of Policy Analysis and Management, 21, April, pp. 191-217. (Dayton, Ohio: Gains for African Americans)

Peterson, Paul E., and William G. Howell. 2004. “Efficiency, Bias, and Classification Schemes: A Response to Alan B. Krueger and Pei Zhu.” American Behavioral Scientist, 47(5): 699-717.  (New York City: Gains for African Americans)

This 1 study concludes that no sub-group of student participants experienced achievement gains from the voucher:

Krueger, Alan B., and Pei Zhu. 2004. “Another Look at the New York City School Voucher Experiment,” The American Behavioral Scientist 47 (5):658–698.

(Update: For a review of systemic effect research — how expanded competition affects achievement in traditional public schools — see here.)

Being Misquoted

July 17, 2008

Dean Millot has a new post attacking me on the peer review issue that Eduwonkette promotes on her own site.

But Dean Millot is being fundamentally dishonest in that he misquotes me. He says that I argue: “In short, I see no problem with research becoming public with little or no review.”

In fact I wrote: “In short, I see no problem with research initially becoming public with little or no review.” (See here )

The absence of the word “initially” makes quite a difference and sets up the straw man that Millot wishes to knock down. The issue is not whether research can benefit from peer review, but whether it is inappropriate to make it publicly available INITIALLY, before it has received peer review.

Readers may want to wonder about the credibility of Millot’s claim that “One of the reasons I do my best to quote the very words of people I write about in edbizbuzz is that I prefer to fight fair.”

And so much for Eduwonkette’s praise of Millot’s “measured, careful, and thoughtful analysis.”

I’m waiting for the correction and apology from both of them.

It Never Ends

July 14, 2008

I thought that the exchange with Eduwonkette over the appropriateness of releasing research without peer review had run its course with my last post.  But it seems that it will never end.  Here is her latest post and here is the reply that I posted in her comment section:

Eduwonkette is attempting to change the subject. I’ve never disputed that peer review can help provide additional assurances to readers about quality.  The issue is whether research ought to be available to the public even if it has not been peer reviewed.  In attacking the release of my most recent study Eduwonkette seems to be arguing that it is inappropriate to release research without peer review, at least under certain conditions that she only applies to research whose findings she does not like.  If she were going to be consistent, she would have to criticize anyone who releases working papers of their research, which would be almost everyone doing serious research.


What’s more, she is still trapped in a contradiction: she can’t say that we should analyze the motives of people who release research directly to the public when assessing whether it is appropriate, while she prevents analysis of her own motives because she blogs anonymously.  As I have now said several times, either she drops the suggestion that we analyze motives or she drops her role as an anonymous blogger.  If she refuses to resolve this contradiction, Ed Week should stop lending her their reputation by hosting her blog.  Let her be inconsistent in blogging at the expense of her own anonymous persona and not drain the respectability of Ed Week.


Lastly, the comparison of the market for education policy information and the market for cars comes from my most recent post in our exchange, but she oddly does not credit me here. (See )  Her position seems to be that we ought to forbid (or at least shun) the sale of used cars without warranties (translation: research without peer review).  My argument is that used cars without warranties come at a risk but there are compensating benefits.  Similarly, non-peer-reviewed research has its risks but also its benefits.


UPDATE — My exchange with Eduwonkette continues although it seems increasingly pointless.  Here is my (slightly edited) last comment on her site:

“Let’s make this very concrete. Was it inappropriate for Marcus Winters and I to release our social promotion findings in 2004 without peer review, or should we have waited until it had been peer-reviewed and published (in various forms) in 2006, 2007, and again in 2008? If the appropriate thing is to wait, would interest groups, editorial boards, and bloggers similarly hold their tongues until the additional evidence came in?  Would policymakers hold off on decisions that might have come out differently if they had the suppressed information?

Would it have been OK to release in 2004 as long as we tried to make it obscure enough so that people were less likely to find it? What if interest groups, bloggers, etc… found our obscure finding and promoted them (as has happened with Jesse Rothstein’s paper)?

And in saying ‘working papers and thinktank reports are released for entirely different functions’ you are repeating your call for an analysis of motives. You’ve said that think tanks want to influence policy (bad motive) while academics are trying to advance knowledge with each other (good motive). But if academics are serving the public good, shouldn’t they ultimately want to influence policy? I am an academic who also releases working papers through a think tank. Does that make my motives good or bad? I think all of this analysis of motives is silly when the real issue is the truth of claims, not why people are making those claims. Calling for an analysis of motives is especially silly for someone who is trying to influence people anonymously. The fact that you are trying to influence people through a blog does not give you a free pass from having to be consistent on this.”


Get every new post delivered to your Inbox.

Join 2,539 other followers