Public Service Announcement: This Study Stinks

(Guest Post by Mike McShane and Gary Ritter)

That’s it, you heard it here first folks, packs of wild dogs have seized control of America’s major cities.

As crazy as that sounds, a study has been circulating the AERA-/Blogo-/twittersphere that’s states that urban Texas school districts have a black male graduation rate of over 80%.  We all know how much some folks here on the Jay P Greene Blog love Texas, but that is just a bit too hard to swallow.

Let’s back up a minute.  Over the past few days, the press (or actually, Diane Ravitch’s twitter page, and then the press, obediently) picked up a story about a “new” study.  OK, it actually isn’t “new” (it first came out in the Berkeley Review of Education in the Fall of 2011) nor is it really a “study”, but more on that later.   The purpose of this study was clear (to attack KIPP) but in the authors’ zeal, they ended up reporting something too good to be true.

Very quickly, using some rough data from schools in Texas, the authors claim to find that charter schools in Texas, and KIPP schools in particular, have higher attrition rates than comparable public schools, even though KIPP schools allegedly spend more money per pupil.

It appears that the authors, in their haste to smear KIPP schools and disprove the strawman idea that choice (as envisioned today) is a “panacea” (using a 20 year old quote) for all of the ills of the American education system, made some pretty shocking errors and omissions that call into question nearly all of their conclusions.

First of all, several of the alleged “findings” were not “found” in this “study”.  Rather, the authors fill their abstract and conclusions with rehashed claims from an earlier, widely discredited study (see this and this and this and this criticism of the flawed Gary Miron paper).

As for the errors in this paper, there are several.  We’ll just highlight a few of the most glaring:

  1.  First, we derive the 80% graduation number from tables 7 and 8 (pg. 169), which report an annual dropout rate from black students of 3% for grades 6-12 in the “comparable urban districts” of Austin, Dallas, and Houston.  Before we dive into the glaring problems of tables 7 and 8, we must first draw attention to the author’s violation of the denominator law.  We don’t know, in the context of this report, what 3% even means.  That is, what is the numerator and what is the denominator that created that rate?  Is that a yearly figure?  Is that a cohort figure?  The authors are absolutely unclear.  Our best guess is that this is a yearly figure, which if compounded, would put the dropout rate for those districts at about 20% for that time period.  As a point of comparison, the dropout rate nationwide for Black males is 53%; if the authors are right, we should all move to the Lone Star State!

 2.  If that is too hard to believe, the tables also report that this 3% figure is lower than the 4% of black dropouts in the rest of the state.  So, if the Texas miracle didn’t do enough to impress you, you can find Texas to be probably the only state where suburban and rural areas have higher dropout rates than cities.

 3.  In addition to farcically large results, tables 7 and 8 (on pg. 169) also appear to have either basic arithmetic mistakes and/or are missing many of their observations when calculating their graduation rates.  The first two columns “Majority black” and “not majority black” should be comprehensive; that is, all of the observations should fall into one of those two categories.  The same is true with the third and fourth column “>100 (Black Students)” and “<100 (Black Students)”. Thus, both of the numbers in the N’s of these columns should sum to the same number.  However, they don’t. In table 7, the first two columns sum to 167 total charter schools, while the second two columns sum to 245 (incidentally the same number as the “All Charters” N).  The same holds true in Table 8, where the first two columns sum to 243, while the second two sum to 373 (again the same number as the “All Charters”).  So where did the other schools go?

Beyond these problems with the author’s primary analyses, this article eschews higher quality studies of the question at hand to focus on clearly flawed research on the topic.  Mathematica already looked into this question in rigorous studies that found positive impacts on achievement, and “did not find levels of attrition among these KIPP middle schools systematically higher (or lower) than those of other “ schools within their districts (they were also clear about the descriptions and sources of the numbers used in the analysis).

In short, any reasonable person who actually read the content of this “new study” would immediately see so many red flags as to take some serious pause before disseminating the findings unqualified to the universe of education news followers.  (We wonder how closely Ms. Ravitch reviewed the study?  She may well have tweeted first and asked questions later!)  Unfortunately, we live in a world populated by many, many, many unreasonable people.

On the bright side, good research continues to show that KIPP schools are effective for underserved students, most serious people disregard “new studies” that are neither “new” nor “studies”, and hard-working KIPP students, teachers, and school leaders keep going about their work each day.

By the way, if you want to see KIPP’s response to this study, it is here.


6 Responses to Public Service Announcement: This Study Stinks

  1. Matthew Ladner says:

    There can be no doubt that the greatness of Texas is lessened by a persistent tendency to create imaginary dropout rates.

    Back about 10 years ago, they were calculating an annual dropout rate starting from 7th grade, and relying on “leaver codes” that had dozens of options other than “Dropped Out.” I recall that one district official spilled the beans after being confronted with data showing that he had coded a large number of students mysteriously enrolling in a Mississippi GED program on the same day.

  2. Greg Forster says:

    Even so, scummy dropout coding is one area where Texas still doesn’t lead the nation. I almost wish Washington state hadn’t discontinued the notorious “dead or other” category, which slaughtered thousands of poor, innocent children every year.

  3. GGW says:


    It took me all of 5 minutes to find a fairly well documented black graduation rate in Texas of 53%.

    Dallas 39%

    Houston 44%

    Of course I have a secret tool called Google. Only available to the 1%.

  4. GGW says:

    The 3% is often a number that districts choose to be Sep-June only. During the school year dropout.

    Of course most dropout happens when kids don’t return to school, ie, summer.

  5. Certainly, KIPP has been successful with its select populations, and that should be praised. And, the opportunity to expand KIPP should be available to any who want it.

    However, public ed critics must remember that not all kids will commit to KIPP standards, and these standards must be chosen by the kids to be successful. KIPP cannot simply be adapted – or forced upon – to the neighborhood school and be successful. The test case for this was in Denver.

    Cole Middle School was the poster child for failure when KIPP leaders agreed to come in and save the school. They failed, and eventually pulled out of their contract with the neighborhood school because the community would not accept, meet, or respond to the model.

    Too many critics think the “panacea” is to simply apply successful charter ideas to all schools. But it doesn’t work that way. KIPP is great for kids who choose it. But not all will. And “choice” is simply not the answer for the rest. Especially, when they choose not to choose.

  6. bbetzen says:

    I had only started to check on Kipp enrollment patterns some weeks ago with a study now online at It is obvious that Kipp Truth Academy certainly is loosing more 5th graders before 8th grade than is DISD. This same study needs to be done with all other Kipp schools, as well as other charter schools. Indications from another study quoted on the above linked page show that there should be other charter schools with similarly high attrition rates in Dallas County. We must look at the numbers!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s