Julian Betts and Emily Tang at the University of California at San Diego have a new systematic review of the research on charter schools. They look at more than 30 studies that meet minimal criteria for research quality. They find that charters have statistically significant positive effects on math and reading achievement in elementary grades and on math in middle school. There are no significant effects for reading in middle school or for high school student achievement. The size of the effects are modest, ranging between 2% and 6% of a standard deviation. (See Table 2)
It’s important to step back and review an entire literature, rather than focus on a single study. It is sensible to focus on higher quality research, since results are highly sensitive to research design. But it is completely inappropriate and misleading to pick a single study while ignoring all others of equal or higher quality simply because that one study produces the result you like.
Of course, highlighting the one study she likes is Diane Ravtich’s stock in trade. All we hear from Ravitch and her Army of Angry Teachers is about the CREDO study. That’s one study — and not a high quality one. And even then Ravitch distorts what CREDO finds.
But Betts and Tang’s review includes CREDO and dozens of other studies. When we look at the full set of research, we see some significant and positive results. And in Table 4, Betts and Tang show us that if you exclude the CREDO study, the positive effects for charters get stronger so that charters significantly improve math achievement across all grades.
Of course, you shouldn’t exclude that one study, but it is informative that the one study that Ravitch and her Army of Angry Teachers hold up as proof of their view that charters don’t work is clearly an outlier from the full set of research. And if we focus on the highest quality random-assignment studies of charter schools, the positive results are even stronger.
I wonder if Diane does this in her historical research. Does she pick the one quotation or document that supports her argument while misleading readers about the entire set of information? It’s harder to catch Ravitch in this sort of deceptive scholarship in historical work, but in quantitative empirical research, it is the essence of what she does.
(edited for typos)
Jay P. Greene's Blog 
It’s worth adding that the laws of statistical analysis teach us to expect the data to produce a wrong result up to 5% of the time. An outlier study is not at all surprising and doesn’t cast any doubt on the consensus finding. This factor makes it all the more irresponsible to cherry-pick the outlier study; given its outlier status, statistical error becomes a much more likely explanation of its finding.
Actually, it looks like virtually no-one has completely read CREDO’s study. Go take a look at the comments about effects of charters over time. They are buried around page 32, but they are there.
Lo and behold, once kids spend enough time in charters to overcome the many problems they bring from public schools (takes about 2 to 3 years), CREDO says charters DO outperform.
It’s not a matter of carter opponents just selecting a report they like, it’s a much more disingenuous matter of not even quoting that report completely.
Read a lot more about this by searching http://www.bipps.org/bipps-blog with the keyword “CREDO.”
[…] this paper has been called “cherrypicking.” Steve Brill sometimes levels this accusation, as do others. It is supposed to imply that CREDO is an exception – that most of the evidence out there finds […]
[…] has been called “cherrypicking.” Steve Brill sometimes levels this accusation, as do others. It is supposed to imply that CREDO is an exception — that most of the evidence out there […]
[…] has been called “cherrypicking.” Steve Brill sometimes levels this accusation, as do others. It is supposed to imply that CREDO is an exception — that most of the evidence out there […]