Jay P. Greene's Blog

(Guest Post by Mike McShane and Gary Ritter)

That’s it, you heard it here first folks, packs of wild dogs have seized control of America’s major cities.

As crazy as that sounds, a study has been circulating the AERA-/Blogo-/twittersphere that’s states that urban Texas school districts have a black male graduation rate of over 80%.  We all know how much some folks here on the Jay P Greene Blog love Texas, but that is just a bit too hard to swallow.

Let’s back up a minute.  Over the past few days, the press (or actually, Diane Ravitch’s twitter page, and then the press, obediently) picked up a story about a “new” study.  OK, it actually isn’t “new” (it first came out in the Berkeley Review of Education in the Fall of 2011) nor is it really a “study”, but more on that later.   The purpose of this study was clear (to attack KIPP) but in the authors’ zeal, they ended up reporting something too good to be true.

Very quickly, using some rough data from schools in Texas, the authors claim to find that charter schools in Texas, and KIPP schools in particular, have higher attrition rates than comparable public schools, even though KIPP schools allegedly spend more money per pupil.

It appears that the authors, in their haste to smear KIPP schools and disprove the strawman idea that choice (as envisioned today) is a “panacea” (using a 20 year old quote) for all of the ills of the American education system, made some pretty shocking errors and omissions that call into question nearly all of their conclusions.

First of all, several of the alleged “findings” were not “found” in this “study”.  Rather, the authors fill their abstract and conclusions with rehashed claims from an earlier, widely discredited study (see this and this and this and this criticism of the flawed Gary Miron paper).

As for the errors in this paper, there are several.  We’ll just highlight a few of the most glaring:

1.  First, we derive the 80% graduation number from tables 7 and 8 (pg. 169), which report an annual dropout rate from black students of 3% for grades 6-12 in the “comparable urban districts” of Austin, Dallas, and Houston.  Before we dive into the glaring problems of tables 7 and 8, we must first draw attention to the author’s violation of the denominator law.  We don’t know, in the context of this report, what 3% even means.  That is, what is the numerator and what is the denominator that created that rate?  Is that a yearly figure?  Is that a cohort figure?  The authors are absolutely unclear.  Our best guess is that this is a yearly figure, which if compounded, would put the dropout rate for those districts at about 20% for that time period.  As a point of comparison, the dropout rate nationwide for Black males is 53%; if the authors are right, we should all move to the Lone Star State!

 2.  If that is too hard to believe, the tables also report that this 3% figure is lower than the 4% of black dropouts in the rest of the state.  So, if the Texas miracle didn’t do enough to impress you, you can find Texas to be probably the only state where suburban and rural areas have higher dropout rates than cities.

 3.  In addition to farcically large results, tables 7 and 8 (on pg. 169) also appear to have either basic arithmetic mistakes and/or are missing many of their observations when calculating their graduation rates.  The first two columns “Majority black” and “not majority black” should be comprehensive; that is, all of the observations should fall into one of those two categories.  The same is true with the third and fourth column “>100 (Black Students)” and “<100 (Black Students)”. Thus, both of the numbers in the N’s of these columns should sum to the same number.  However, they don’t. In table 7, the first two columns sum to 167 total charter schools, while the second two columns sum to 245 (incidentally the same number as the “All Charters” N).  The same holds true in Table 8, where the first two columns sum to 243, while the second two sum to 373 (again the same number as the “All Charters”).  So where did the other schools go?

Beyond these problems with the author’s primary analyses, this article eschews higher quality studies of the question at hand to focus on clearly flawed research on the topic.  Mathematica already looked into this question in rigorous studies that found positive impacts on achievement, and “did not find levels of attrition among these KIPP middle schools systematically higher (or lower) than those of other “ schools within their districts (they were also clear about the descriptions and sources of the numbers used in the analysis).

In short, any reasonable person who actually read the content of this “new study” would immediately see so many red flags as to take some serious pause before disseminating the findings unqualified to the universe of education news followers.  (We wonder how closely Ms. Ravitch reviewed the study?  She may well have tweeted first and asked questions later!)  Unfortunately, we live in a world populated by many, many, many unreasonable people.

On the bright side, good research continues to show that KIPP schools are effective for underserved students, most serious people disregard “new studies” that are neither “new” nor “studies”, and hard-working KIPP students, teachers, and school leaders keep going about their work each day.

By the way, if you want to see KIPP’s response to this study, it is here.

I don’t think Marc Tucker liked my review of his book in Education Next.  He responds in this post on the Ed Next blog.  And my reply to his response is also on the Ed Next blog.

Above I’ve posted a video of Tucker explaining his response.

I received this amazing press release from Tom Vander Ark about how computer grading of essays may be as accurate as human grading.  I’m not sure if this means that computer grading has really advanced or if human grading really stinks.  Besides, I don’t even know why the scientists invented the robots.

In any event, here is the release:

A direct comparison between human graders and software designed to score student essays achieved virtually identical levels of accuracy, with the software in some cases proving to be more reliable, a groundbreaking study has found.

The study, which was underwritten by the William and Flora Hewlett Foundation and conducted by experts in educational measurement and assessment, will be released here on Monday, April 16th, at the annual conference of the National Council on Measurement in Education. An advance copy of the study is available today at http://bit.ly/HJWwdP.

“The demonstration showed conclusively that automated essay scoring systems are fast, accurate, and cost effective,” said Tom Vander Ark, CEO of Open Education Solutions, which provides consulting serves related to digital learning, and co-director of the study.

That’s important because writing essays are one important way for students to learn critical reasoning, but teachers don’t assign them often enough because grading them is both expensive and time consuming. Automated scoring of essays holds the promise of lowering the cost and time of having students write so they can do it more often.

Education experts believe that critical reasoning and writing are part of a suite of skills that students need to be competitive in the 21st century. Others are working collaboratively, communicating effectively and learning how to learn, as well as mastering core academic content. The Hewlett Foundation calls this suite of skills Deeper Learning and is making grants to encourage its adoption at schools throughout the country.

“Better tests support better learning,” says Barbara Chow, Education Program Director at the Hewlett Foundation. “This demonstration of rapid and accurate automated essay scoring will encourage states to include more writing in their state assessments. And, the more we can use essays to assess what students have learned, the greater the likelihood they’ll master important academic content, critical thinking, and effective communication.”

For more than 20 years, companies that provide automated essay scoring software have claimed that their systems can perform as effectively, more affordably and faster than other available methods of essay scoring. The study was the first comprehensive multi-vendor trial to test those claims. The study challenged nine companies that constitute more than ninety-seven percent of the current market of commercial providers of automated essay scoring to compare capabilities. More than 16,000 essays were released from six participating state departments of education, with each set of essays varying in length, type, and grading protocols. The essays were already hand scored according to state standards. The challenge was for companies to approximate established scores by using software.

At a time when the U.S. Department of Education is funding states to design and develop new forms of high-stakes testing, the study introduces important data. Many states are limited to multiple-choice formats, because more sophisticated measures of academic performance cost too much to grade and take too long to process. Forty-five states are already actively overhauling testing standards, and many are considering the use of machine scoring systems.

The study grows from a contest call the Automated Student Assessment Prize, or ASAP, which the Hewlett Foundation is sponsoring to evaluate the current state of automated testing and to encourage further developments in the field.

In addition to looking at commercial vendors, the contest is offering $100,000 in cash prizes in a competition open to anyone to develop new automated essay scoring techniques. The open competition is underway now and scheduled to close on April 30th. The pool of $100,000 will be awarded the best performers. Details of the public competition are available atwww.kaggle.com/c/ASAP-AES . The open competition website includes an active leader board to document prize rules, regularly updated results, and discussion threads between competitors.

The goal of ASAP is to offer a series of impartial competitions in which a fair, open and transparent participation process will allow key participants in the world of education and testing to understand the value of automated student assessment technologies.

ASAP is being conducted with the support of the Partnership for Assessment of Readiness for College and Careers and Smarter Balanced Assessment Consortium, two multi-state consortia funded by the U.S. Department of Education to develop next-generation assessments. ASAP is aligned with the aspirations of the Common Core State Standards and seeks to accelerate assessment innovation to help more students graduate from college and to become career ready.

Jaison Morgan, CEO of The Common Pool, a consulting business that specializes in developing effective incentive models for solving problems, and co-director of the study, said the prize and studies will raise broader awareness of the current capabilities of automated scoring of essays.

“By offering a private demonstration of current capabilities, we can reveal to our state partners what is already commercially available,” Morgan said. “But, by complimenting it with a public competition, we will attract new participants to the field and investment from new players. We believe that the public competition will trigger major breakthroughs.”

ASAP is preparing to introduce a second study, in which private providers and public competitors will be challenged to reveal the capabilities of automated scoring systems for grading short-answer questions. The second study will be conducted this summer. There are another three ASAP studies in development.