(Guest Post by Mike McShane and Gary Ritter)
That’s it, you heard it here first folks, packs of wild dogs have seized control of America’s major cities.
As crazy as that sounds, a study has been circulating the AERA-/Blogo-/twittersphere that’s states that urban Texas school districts have a black male graduation rate of over 80%. We all know how much some folks here on the Jay P Greene Blog love Texas, but that is just a bit too hard to swallow.
Let’s back up a minute. Over the past few days, the press (or actually, Diane Ravitch’s twitter page, and then the press, obediently) picked up a story about a “new” study. OK, it actually isn’t “new” (it first came out in the Berkeley Review of Education in the Fall of 2011) nor is it really a “study”, but more on that later. The purpose of this study was clear (to attack KIPP) but in the authors’ zeal, they ended up reporting something too good to be true.
Very quickly, using some rough data from schools in Texas, the authors claim to find that charter schools in Texas, and KIPP schools in particular, have higher attrition rates than comparable public schools, even though KIPP schools allegedly spend more money per pupil.
It appears that the authors, in their haste to smear KIPP schools and disprove the strawman idea that choice (as envisioned today) is a “panacea” (using a 20 year old quote) for all of the ills of the American education system, made some pretty shocking errors and omissions that call into question nearly all of their conclusions.
First of all, several of the alleged “findings” were not “found” in this “study”. Rather, the authors fill their abstract and conclusions with rehashed claims from an earlier, widely discredited study (see this and this and this and this criticism of the flawed Gary Miron paper).
As for the errors in this paper, there are several. We’ll just highlight a few of the most glaring:
- First, we derive the 80% graduation number from tables 7 and 8 (pg. 169), which report an annual dropout rate from black students of 3% for grades 6-12 in the “comparable urban districts” of Austin, Dallas, and Houston. Before we dive into the glaring problems of tables 7 and 8, we must first draw attention to the author’s violation of the denominator law. We don’t know, in the context of this report, what 3% even means. That is, what is the numerator and what is the denominator that created that rate? Is that a yearly figure? Is that a cohort figure? The authors are absolutely unclear. Our best guess is that this is a yearly figure, which if compounded, would put the dropout rate for those districts at about 20% for that time period. As a point of comparison, the dropout rate nationwide for Black males is 53%; if the authors are right, we should all move to the Lone Star State!
2. If that is too hard to believe, the tables also report that this 3% figure is lower than the 4% of black dropouts in the rest of the state. So, if the Texas miracle didn’t do enough to impress you, you can find Texas to be probably the only state where suburban and rural areas have higher dropout rates than cities.
3. In addition to farcically large results, tables 7 and 8 (on pg. 169) also appear to have either basic arithmetic mistakes and/or are missing many of their observations when calculating their graduation rates. The first two columns “Majority black” and “not majority black” should be comprehensive; that is, all of the observations should fall into one of those two categories. The same is true with the third and fourth column “>100 (Black Students)” and “<100 (Black Students)”. Thus, both of the numbers in the N’s of these columns should sum to the same number. However, they don’t. In table 7, the first two columns sum to 167 total charter schools, while the second two columns sum to 245 (incidentally the same number as the “All Charters” N). The same holds true in Table 8, where the first two columns sum to 243, while the second two sum to 373 (again the same number as the “All Charters”). So where did the other schools go?
Beyond these problems with the author’s primary analyses, this article eschews higher quality studies of the question at hand to focus on clearly flawed research on the topic. Mathematica already looked into this question in rigorous studies that found positive impacts on achievement, and “did not find levels of attrition among these KIPP middle schools systematically higher (or lower) than those of other “ schools within their districts (they were also clear about the descriptions and sources of the numbers used in the analysis).
In short, any reasonable person who actually read the content of this “new study” would immediately see so many red flags as to take some serious pause before disseminating the findings unqualified to the universe of education news followers. (We wonder how closely Ms. Ravitch reviewed the study? She may well have tweeted first and asked questions later!) Unfortunately, we live in a world populated by many, many, many unreasonable people.
On the bright side, good research continues to show that KIPP schools are effective for underserved students, most serious people disregard “new studies” that are neither “new” nor “studies”, and hard-working KIPP students, teachers, and school leaders keep going about their work each day.
By the way, if you want to see KIPP’s response to this study, it is here.