Jay P. Greene's Blog

Diane Ravitch’s new book “The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education” has been burning up the charts. Ravitch has been ubiquitous, writing op-eds in support of her book, doing lectures and interviews all over the place, and being reviewed in all sorts of high-profile venues.

As an overall matter, the book says little, if anything, that is actually new on the subjects of testing and choice. What Ravitch is really selling with this book is the story of her personal and ideological conversion. Not so long ago, she was writing articles like “In Defense of Testing,” or “The Right Thing: Why Liberals Should Be Pro-Choice,” a lengthy article in The New Republic that remains one of the most passionate and eloquent defenses of school choice and vouchers in particular. Now she seems to be a diehard opponent of these things. But she’s not saying anything that other diehard opponents haven’t already said countless times.

The book does score a few points in critiquing the charter school movement (e.g., charter schools have an unfair advantage in competing with Catholic schools in the inner cities, and charter test results haven’t been as promising as might have been expected), or in critiquing testing and accountability (e.g., states have been watering down their standards, as shown by wide discrepancies between NAEP and state tests).

But these few good points are outweighed by the bad arguments and leaps of illogic that permeate much of the book. The book’s faults fall into five general categories, each of which will be the subject of a blog post this week:

1. Ignoring or selectively citing scholarly literature;
2. Misinterpreting the scholarly literature that she does cite;
3. Caricaturing her opponents in terms of strawman arguments, rather than taking the best arguments head-on;
4. Tendering logical fallacies; and
5. Engaging in a double standard, such as holding a disfavored position to a high burden of proof while blithely accepting more problematic evidence that supports one’s own position (or not looking for evidence at all).

IGNORING SCHOLARLY LITERATURE

An endemic problem with Ravitch’s book is the tendency to cite only one or two studies on a disputed empirical question as if that settled the matter, while ignoring other (often better) studies that undermine or refute her claims.

For example, Ravitch claims that vouchers don’t pressure traditional public school systems to improve (pp. 129-32), even though the scholarly consensus is precisely the opposite. Ravitch also highlights a couple of studies that failed to find achievement gains from vouchers, but ignores the fact that “9 of the 10 [random assignment studies] show significant, positive effects for at least some subgroups of students.“

One of the most egregious examples arises from Ravitch’s repetitive claim that charter schools tap into the most “motivated” students. This claim appears practically every time Ravitch mentions charter schools. See, e.g., p. 145 (“charter schools are havens for the motivated”); p. 156 (“A lottery for admission tends to eliminate unmotivated students”); p. 212 (“two-tiered system in urban districts, with charter schools for motivated students and public schools for all those left behind”); p. 220 (“Charter schools in urban centers will enroll the motivated children of the poor, while the regular public schools will become schools of last resort for those who never applied or were rejected.”); p. 227 (“Our schools cannot improve if charter schools siphon away the most motivated students”).

Notably, Ravitch doesn’t highlight any actual evidence for this claim. She treats it as definitionally true (“by definition, only the most motivated families apply for a slot,” p. 135). But that is wrong: The only thing that could be true by definition here is that parents who sign up their children for charter schools are the most motivated to sign up their children for charter schools, which is a trivial observation (and one that probably isn’t true anyway: some motivated parents might easily fail to hear about a charter school opportunity, while other parents might sign up on a whim).

But that’s not the “motivation” that Ravitch means. What Ravitch tries to imply — and what she lacks any evidence for — is that charter schools all over the country are over-enrolling those students who are the most motivated to succeed academically. That’s the only thing that could possibly lead to an unfair charter school advantage. To be sure, there are undoubtedly some charter students who are the most academically well-prepared and who are leaving the public school to seek a greener pasture elsewhere. But, Ravitch has zero evidence that these children are in the majority.

Nor would such a contention be consistent with the actual evidence, which Ravitch doesn’t bother to investigate (having presumed to settle the motivation issue “by definition”). In fact, a recent paper by Zimmer et al. analyzed data “from states that encompass about 45 percent of all charter schools in the nation.” They found: “Students transferring to charter schools had prior achievement levels that were generally similar to or lower than those of their [traditional public school] peers. And transfers had surprisingly little effect on racial distributions across the sites.” Similarly, Booker, Zimmer, and Buddin (2005) found that in California and Texas — both huge charter states — students who transferred to charter schools had lower test scores than their peers at public schools.

Given this evidence, it is more plausible to suspect that many charter school entrants have been struggling to get by in the public school, and they (or their parents) are “motivated” only in the sense that they’re trying to find something that might work. It’s hard to see how that sort of motivation would create an unfair advantage on the part of charter schools, as Ravitch wants the reader to believe.

There are numerous other examples of Ravitch ignoring scholarly literature that she finds inconvenient:

1. Ravitch focuses on a few studies about whether charter schools increase test scores. Leaving aside the fact that this is completely incoherent (given that Ravitch’s whole point elsewhere is that test scores shouldn’t be used to tell us the worth of a school), Ravitch ignores the recent study showing that charter schools increased the likelihood that a student will graduate and go to college. These are worthy goals.

2. Ravitch cites Walt Haney’s study asserting that “dramatic gains in Texas on its state tests” were a myth. (p. 96). But she ignores the Toenjes/Dworkin article contending that Haney’s article was biased and unreliable.

3. Ravitch attacks NCLB for failing to bring about its intended goal: improved test scores. For this argument, she relies on snapshots of NAEP scores during the 2000s. (pp. 109-10). But one looks in vain for Ravitch to cite Hanushek and Raymond’s paper noting that it is “not possible to investigate the impact of NCLB directly” — that is, it is not possible to do exactly what Ravitch purported to do. This is because “the majority of states had already instituted some sort of accountability system by the time the federal law took effect . . . 39 states did so by 2000.”

Hanushek and Raymond went on to find that “the introduction of accountability systems into a state tends to lead to larger achievement growth than would have occurred without accountability. The analysis, however, indicates that just reporting results has minimal impact on student performance and that the force of accountability comes from attaching consequences such as monetary awards or takeover threats to school performance. This finding supports the contested provisions of NCLB that impose sanctions on failing schools.” This finding is similar to Carnoy and Loeb 2002 (another paper left uncited by Ravitch), who found that “students in high-accountability states averaged significantly greater gains on the NAEP 8th-grade math test than students in states with little or no state measures to improve student performance.”