Diane Ravitch’s new book “The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education” has been burning up the charts. Ravitch has been ubiquitous, writing op-eds in support of her book, doing lectures and interviews all over the place, and being reviewed in all sorts of high-profile venues.
As an overall matter, the book says little, if anything, that is actually new on the subjects of testing and choice. What Ravitch is really selling with this book is the story of her personal and ideological conversion. Not so long ago, she was writing articles like “In Defense of Testing,” or “The Right Thing: Why Liberals Should Be Pro-Choice,” a lengthy article in The New Republic that remains one of the most passionate and eloquent defenses of school choice and vouchers in particular. Now she seems to be a diehard opponent of these things. But she’s not saying anything that other diehard opponents haven’t already said countless times.
The book does score a few points in critiquing the charter school movement (e.g., charter schools have an unfair advantage in competing with Catholic schools in the inner cities, and charter test results haven’t been as promising as might have been expected), or in critiquing testing and accountability (e.g., states have been watering down their standards, as shown by wide discrepancies between NAEP and state tests).
But these few good points are outweighed by the bad arguments and leaps of illogic that permeate much of the book. The book’s faults fall into five general categories, each of which will be the subject of a blog post this week:
- Ignoring or selectively citing scholarly literature;
- Misinterpreting the scholarly literature that she does cite;
- Caricaturing her opponents in terms of strawman arguments, rather than taking the best arguments head-on;
- Tendering logical fallacies; and
- Engaging in a double standard, such as holding a disfavored position to a high burden of proof while blithely accepting more problematic evidence that supports one’s own position (or not looking for evidence at all).
IGNORING SCHOLARLY LITERATURE
An endemic problem with Ravitch’s book is the tendency to cite only one or two studies on a disputed empirical question as if that settled the matter, while ignoring other (often better) studies that undermine or refute her claims.
For example, Ravitch claims that vouchers don’t pressure traditional public school systems to improve (pp. 129-32), even though the scholarly consensus is precisely the opposite. Ravitch also highlights a couple of studies that failed to find achievement gains from vouchers, but ignores the fact that “9 of the 10 [random assignment studies] show significant, positive effects for at least some subgroups of students.“
One of the most egregious examples arises from Ravitch’s repetitive claim that charter schools tap into the most “motivated” students. This claim appears practically every time Ravitch mentions charter schools. See, e.g., p. 145 (“charter schools are havens for the motivated”); p. 156 (“A lottery for admission tends to eliminate unmotivated students”); p. 212 (“two-tiered system in urban districts, with charter schools for motivated students and public schools for all those left behind”); p. 220 (“Charter schools in urban centers will enroll the motivated children of the poor, while the regular public schools will become schools of last resort for those who never applied or were rejected.”); p. 227 (“Our schools cannot improve if charter schools siphon away the most motivated students”).
Notably, Ravitch doesn’t highlight any actual evidence for this claim. She treats it as definitionally true (“by definition, only the most motivated families apply for a slot,” p. 135). But that is wrong: The only thing that could be true by definition here is that parents who sign up their children for charter schools are the most motivated to sign up their children for charter schools, which is a trivial observation (and one that probably isn’t true anyway: some motivated parents might easily fail to hear about a charter school opportunity, while other parents might sign up on a whim).
But that’s not the “motivation” that Ravitch means. What Ravitch tries to imply — and what she lacks any evidence for — is that charter schools all over the country are over-enrolling those students who are the most motivated to succeed academically. That’s the only thing that could possibly lead to an unfair charter school advantage. To be sure, there are undoubtedly some charter students who are the most academically well-prepared and who are leaving the public school to seek a greener pasture elsewhere. But, Ravitch has zero evidence that these children are in the majority.
Nor would such a contention be consistent with the actual evidence, which Ravitch doesn’t bother to investigate (having presumed to settle the motivation issue “by definition”). In fact, a recent paper by Zimmer et al. analyzed data “from states that encompass about 45 percent of all charter schools in the nation.” They found: “Students transferring to charter schools had prior achievement levels that were generally similar to or lower than those of their [traditional public school] peers. And transfers had surprisingly little effect on racial distributions across the sites.” Similarly, Booker, Zimmer, and Buddin (2005) found that in California and Texas — both huge charter states — students who transferred to charter schools had lower test scores than their peers at public schools.
Given this evidence, it is more plausible to suspect that many charter school entrants have been struggling to get by in the public school, and they (or their parents) are “motivated” only in the sense that they’re trying to find something that might work. It’s hard to see how that sort of motivation would create an unfair advantage on the part of charter schools, as Ravitch wants the reader to believe.
There are numerous other examples of Ravitch ignoring scholarly literature that she finds inconvenient:
1. Ravitch focuses on a few studies about whether charter schools increase test scores. Leaving aside the fact that this is completely incoherent (given that Ravitch’s whole point elsewhere is that test scores shouldn’t be used to tell us the worth of a school), Ravitch ignores the recent study showing that charter schools increased the likelihood that a student will graduate and go to college. These are worthy goals.
2. Ravitch cites Walt Haney’s study asserting that “dramatic gains in Texas on its state tests” were a myth. (p. 96). But she ignores the Toenjes/Dworkin article contending that Haney’s article was biased and unreliable.
3. Ravitch attacks NCLB for failing to bring about its intended goal: improved test scores. For this argument, she relies on snapshots of NAEP scores during the 2000s. (pp. 109-10). But one looks in vain for Ravitch to cite Hanushek and Raymond’s paper noting that it is “not possible to investigate the impact of NCLB directly” — that is, it is not possible to do exactly what Ravitch purported to do. This is because “the majority of states had already instituted some sort of accountability system by the time the federal law took effect . . . 39 states did so by 2000.”
Hanushek and Raymond went on to find that “the introduction of accountability systems into a state tends to lead to larger achievement growth than would have occurred without accountability. The analysis, however, indicates that just reporting results has minimal impact on student performance and that the force of accountability comes from attaching consequences such as monetary awards or takeover threats to school performance. This finding supports the contested provisions of NCLB that impose sanctions on failing schools.” This finding is similar to Carnoy and Loeb 2002 (another paper left uncited by Ravitch), who found that “students in high-accountability states averaged significantly greater gains on the NAEP 8th-grade math test than students in states with little or no state measures to improve student performance.”
Jay P. Greene's Blog 
Having been in the private and public school system, here is why I believe the private schools do a better job.
First the money is tied to the PARENTS. Private schools are not perfect, and there are some I would never allow my child to attend, but overall I have a “voice”.
If I don’t like how things are going, I remove my child and my money comes with me. If I’m the only parent, it might not make a big difference, however when there are many parents removing their children, it does speak volumes.
I find an organization that is far more willing to listen. Is it perfect? Of course not but in general we’ve found a better product.
Private schools still test their children, but those test results are given to parents and if they don’t like the results, again, they can pull their kids out and take the money with them.
Not a perfect system, but a better one for sure.
[…] Ignoring or selectively citing scholarly literature; […]
It appears to me that you are guilty of the same sins that you accuse Diane Ravitch of committing. In all you blog posts about vouchers and charters, you “selectively cite” studies which support you position, and you ignore other studies that are inconvenient. You also misinterpret the literature to support you positions. Honestly, it is clear that you support vouchers and charter schools and she does not. Both sides throw studies out there, because in education for each study supporting one side, there are those who do not support it. One of the reasons those of us who work on the front lines in the classrooms and schools get tired of the educational rhetoric coming out of both Washington and Schools of Education is that both are guilty to “pushing the latest fad.” Perhaps we should try the fad you a proposing just so we can move on to the next one.
Hi John,
Stuart has carefully documented examples of how Ravitch has mis-read or ignored research. If you think we’ve done the same, then you need to document that rather than simply assert it.
I would love to examine the charter school research more in depth. What 8 to 10 studies would you recommend I look at? Honestly, I am not convinced one way or the other, but would like to see what the research says.
I would start with this: https://jaypgreene.com/2009/09/22/more-charter-evidence/ and
Follow the links.
Thanks. I was hoping to see some peer-reviewed studies published by folks who are neutral on the issue.
John,
Which studies lacked peer review or “neutrality”? Hoxby and Rockoff’s piece was peer reviewed by NBER, which is one of the leading organization of economists in the country. Booker, Sass, and Zimmer’s study was peer reviewed and published in Education Next, which Education Week ranked as the most influential journal. It was also peer reviewed when it was released by RAND. The Angrist and Kane piece is new but I’ll bet it will appear in a journal soon.
And these people are some of the leading economists in the country from MIT, Harvard, and Stanford. Are they all biased researchers?
This is the second time, John, that you’ve made sweeping claims with no evidence to support them. That may meet Diane Ravitch’s standards, but around here we expect people to back up their claims with evidence.
I honestly do not know much about NBER other than what it states on its web site. Would love to know more about its funding sources. However, the publication Education Next clearly has a partisan bias and is funded in part by organizations who have a clear stake in the voucher outcomes agenda. ( Hoover Institute and Thomas B Fordham, both which have their own agendas in education and can hardly be said to fund unbiased research.)Being influential does not necessarily mean reliable, so the declaration by Education Week is actually meaningless. Rand I recognize and know about them. As far as leading economists in the country, I respect what they know, but by being leading economists that does not mean I accept what they say at face value. Leading economists can be wrong. I do respect independent research that does not have the appearance of being funded by organizations and sources with a known agenda. it does not appear to be the case with NBER and Rand, but to brag about being published in Education Next is not I would call convincing. I believe in searching for the sources of money when it comes to research. Those sources can tell you a great deal about its reliability these days.
John — Can I suggest that you actually read and comment on the merits of the studies rather than analyze their motives?
[…] Ignoring or selectively citing scholarly literature; […]
“but ignores the fact that “9 of the 10 [random assignment studies] show significant, positive effects for at least some subgroups of students.“”
You really don’t do yourself any favours with this sort of claim.
Even if there is no positive effect overall we would expect, just by random variation, some significant positive effects for at least some subgroups of students. Perhaps you mean more than this, but as it stands the fact you refer to would be worth ignoring.
Teachingbattleground — You should follow the link and then you’ll see that 6 random-assignment studies show benefits for all students, 3 for a significant sub-group (like African-Americans, and only one shows no benefit (and no harm). If there really were no benefit from vouchers you wouldn’t get that pattern by chance.
The point I was making is that, statistically speaking, sub-group analyses are not reliable indicators of effectiveness except under very limited circumstances.
Fair enough if you are not relying on these to make your point, but by including them in order to get a 9 out of 10 figure you weaken, not strengthen, your point (at least with people who know how manipulation of data works).
Incidentally, having followed your link, I notice that some of the studies are replications of each other. This doesn’t help your case either as this is also a standard trick for manipulating data. (This is not to say your conclusions about what the studies show are wrong, just that your methodology is suspect on two counts).
You are mistaken on both counts, teachingbattleground. Sub-group analyses are valid and reliable so long as you are not doing so many and not doing one’s for which there is no theortical basis. All of the studies listed only do a handful of sub-group analyses and almost all are limited to racial/ethnic and/or income/disadvantage break-outs. We have strong theoretical reasons to expect that the effects of choice might be different for these sub-groups.
Second, the fact that some of the 10 studies are replications is a GOOD thing. It means that multiple teams of reaserchers have confirmed findings so that we don’t have to suspect manipulation of the data or faulty methods. The fact that multiple teams do the same sub-group break-outs also helps confirm that that is a reasonable reasearch approach. If there weren’t replications you might be complaining that no one has ever confirmed these results. And if you think replications reduce the number of experiments, name me one other education policy that has been subject to more unique random assignment experiments.
You seem to be arguing against vague general principles (“Thou shalt not do a sub-group analysis” or “Experiments must never be replicated”) which condemn the studies. I’m afraid these are strawmen. The methodological error is not in the studies but in how you have counted them up.
All I am saying is that if you want to ascertain the effectiveness of a particular course of action by counting up studies, you do not count subgroup analysis results and you do not count replicated studies, in order to produce a single figure and repeat it as a fact. Both are well known tricks-of-the-trade, mentioned in Ben Goldacre’s “Bad Science” book as something pharmaceutical companies do to sell their products.
Well, I take Jay’s side rather than TB’s, but even if I didn’t, there would still be this point to consider: the empirical evidence for vouchers is stronger by several orders of magnitude than the empirical evidence supporting any other education reform policy. So unless we’re prepared to say that we really have no reliable data on anything, in which case we should all quit talking about education reform and go do something else, we can’t escape the conclusion that vouchers are by far the most promising approach to education reform.
Well, if you want to take sides on the methodological point it would be nice if you presented an actual argument.
As to the other point, aren’t you just saying that of all the magic bullets invented so far, vouchers is the least discredited? That hardly counters the argument that there are no magic bullets.
TB — I’m not sure that we really need your “advice” about how best to frame the evidence on vouchers. The substantive issue is whether there is a fairly large body of evidence from rigorous studies that consistently show benefits for students who receive vouchers. I believe there is. What do you believe?
“The substantive issue is whether there is a fairly large body of evidence from rigorous studies that consistently show benefits for students who receive vouchers. I believe there is. What do you believe?”
I believe that you haven’t shown this.
I see. And what education policy has sufficient support to meet your standards? If nothing, how about health policy or any other kind of policy?
I don’t really have an opinion on the US system, as I still have a lot to learn about it. For my own country, my priorities appear at the end of this post:
I’m not really sure what this has to do with anything though. I am not claiming that my preferred policies all have research evidence behind them either.
Peer reviewed oncology drug research combined with full disclosure of any and all financial connections would be one possibility. Granted, such an approach isn’t bulletproof either, but its qualitatively better than what currently passes for evidence supporting one view or another in education.
Just found a really good explanation of the problem with sub-group analysis:
http://www.badscience.net/2009/04/a-frankly-thin-contrivance-for-writing-on-the-fascinating-issue-of-subgroup-analysis/
[…] of students in traditional schools, is conveniently not mentioned by Ravitch in her piece.) And Stuart Buck, among others, has offered his own comprehensive review of Ravitch’s faulty thinking and data […]
[…] as proven by Stuart Buck, her recent tome, The Death and Life of the Great American School System, is merely a volume of […]
[…] teacher Darren Miller, surmising what Dropout Nation and others have been saying about the once-respectable education […]
[…] In reviewing her book, Jay P. Greene lists her sins this way: The book’s faults fall into five general categories, each of which will be the subject of a blog post this week: […]
[…] And while we are talking reactionaries, perhaps their poster child should be Diane Ravitch. She was a liberal before she became conservative before she became a progressive, but she’s really a reactionary. (She is wrong about so much that she should get an award for “Yes I can make this sh*t up.”) In fact, researcher Jay Greene set aside an entire week on his blog to debunk her endless reactionary blather, which he titled “Ravitch is Wrong Week.” […]
[…] And while we are talking reactionaries, perhaps their poster child should be Diane Ravitch. She was a liberal before she became conservative before she became a progressive, but she’s really a reactionary. (She is wrong about so much that she should get an award for “Yes I can make this sh*t up.”) In fact, researcher Jay Greene set aside an entire week on his blog to debunk her endless reactionary blather, which he titled “Ravitch is Wrong Week.” […]