Jay P. Greene's Blog

(Guest Post by Mike McShane and Gary Ritter)

That’s it, you heard it here first folks, packs of wild dogs have seized control of America’s major cities.

As crazy as that sounds, a study has been circulating the AERA-/Blogo-/twittersphere that’s states that urban Texas school districts have a black male graduation rate of over 80%.  We all know how much some folks here on the Jay P Greene Blog love Texas, but that is just a bit too hard to swallow.

Let’s back up a minute.  Over the past few days, the press (or actually, Diane Ravitch’s twitter page, and then the press, obediently) picked up a story about a “new” study.  OK, it actually isn’t “new” (it first came out in the Berkeley Review of Education in the Fall of 2011) nor is it really a “study”, but more on that later.   The purpose of this study was clear (to attack KIPP) but in the authors’ zeal, they ended up reporting something too good to be true.

Very quickly, using some rough data from schools in Texas, the authors claim to find that charter schools in Texas, and KIPP schools in particular, have higher attrition rates than comparable public schools, even though KIPP schools allegedly spend more money per pupil.

It appears that the authors, in their haste to smear KIPP schools and disprove the strawman idea that choice (as envisioned today) is a “panacea” (using a 20 year old quote) for all of the ills of the American education system, made some pretty shocking errors and omissions that call into question nearly all of their conclusions.

First of all, several of the alleged “findings” were not “found” in this “study”.  Rather, the authors fill their abstract and conclusions with rehashed claims from an earlier, widely discredited study (see this and this and this and this criticism of the flawed Gary Miron paper).

As for the errors in this paper, there are several.  We’ll just highlight a few of the most glaring:

1.  First, we derive the 80% graduation number from tables 7 and 8 (pg. 169), which report an annual dropout rate from black students of 3% for grades 6-12 in the “comparable urban districts” of Austin, Dallas, and Houston.  Before we dive into the glaring problems of tables 7 and 8, we must first draw attention to the author’s violation of the denominator law.  We don’t know, in the context of this report, what 3% even means.  That is, what is the numerator and what is the denominator that created that rate?  Is that a yearly figure?  Is that a cohort figure?  The authors are absolutely unclear.  Our best guess is that this is a yearly figure, which if compounded, would put the dropout rate for those districts at about 20% for that time period.  As a point of comparison, the dropout rate nationwide for Black males is 53%; if the authors are right, we should all move to the Lone Star State!

 2.  If that is too hard to believe, the tables also report that this 3% figure is lower than the 4% of black dropouts in the rest of the state.  So, if the Texas miracle didn’t do enough to impress you, you can find Texas to be probably the only state where suburban and rural areas have higher dropout rates than cities.

 3.  In addition to farcically large results, tables 7 and 8 (on pg. 169) also appear to have either basic arithmetic mistakes and/or are missing many of their observations when calculating their graduation rates.  The first two columns “Majority black” and “not majority black” should be comprehensive; that is, all of the observations should fall into one of those two categories.  The same is true with the third and fourth column “>100 (Black Students)” and “<100 (Black Students)”. Thus, both of the numbers in the N’s of these columns should sum to the same number.  However, they don’t. In table 7, the first two columns sum to 167 total charter schools, while the second two columns sum to 245 (incidentally the same number as the “All Charters” N).  The same holds true in Table 8, where the first two columns sum to 243, while the second two sum to 373 (again the same number as the “All Charters”).  So where did the other schools go?

Beyond these problems with the author’s primary analyses, this article eschews higher quality studies of the question at hand to focus on clearly flawed research on the topic.  Mathematica already looked into this question in rigorous studies that found positive impacts on achievement, and “did not find levels of attrition among these KIPP middle schools systematically higher (or lower) than those of other “ schools within their districts (they were also clear about the descriptions and sources of the numbers used in the analysis).

In short, any reasonable person who actually read the content of this “new study” would immediately see so many red flags as to take some serious pause before disseminating the findings unqualified to the universe of education news followers.  (We wonder how closely Ms. Ravitch reviewed the study?  She may well have tweeted first and asked questions later!)  Unfortunately, we live in a world populated by many, many, many unreasonable people.

On the bright side, good research continues to show that KIPP schools are effective for underserved students, most serious people disregard “new studies” that are neither “new” nor “studies”, and hard-working KIPP students, teachers, and school leaders keep going about their work each day.

By the way, if you want to see KIPP’s response to this study, it is here.

I don’t think Marc Tucker liked my review of his book in Education Next.  He responds in this post on the Ed Next blog.  And my reply to his response is also on the Ed Next blog.

Above I’ve posted a video of Tucker explaining his response.

I received this amazing press release from Tom Vander Ark about how computer grading of essays may be as accurate as human grading.  I’m not sure if this means that computer grading has really advanced or if human grading really stinks.  Besides, I don’t even know why the scientists invented the robots.

In any event, here is the release:

A direct comparison between human graders and software designed to score student essays achieved virtually identical levels of accuracy, with the software in some cases proving to be more reliable, a groundbreaking study has found.

The study, which was underwritten by the William and Flora Hewlett Foundation and conducted by experts in educational measurement and assessment, will be released here on Monday, April 16th, at the annual conference of the National Council on Measurement in Education. An advance copy of the study is available today at http://bit.ly/HJWwdP.

“The demonstration showed conclusively that automated essay scoring systems are fast, accurate, and cost effective,” said Tom Vander Ark, CEO of Open Education Solutions, which provides consulting serves related to digital learning, and co-director of the study.

That’s important because writing essays are one important way for students to learn critical reasoning, but teachers don’t assign them often enough because grading them is both expensive and time consuming. Automated scoring of essays holds the promise of lowering the cost and time of having students write so they can do it more often.

Education experts believe that critical reasoning and writing are part of a suite of skills that students need to be competitive in the 21st century. Others are working collaboratively, communicating effectively and learning how to learn, as well as mastering core academic content. The Hewlett Foundation calls this suite of skills Deeper Learning and is making grants to encourage its adoption at schools throughout the country.

“Better tests support better learning,” says Barbara Chow, Education Program Director at the Hewlett Foundation. “This demonstration of rapid and accurate automated essay scoring will encourage states to include more writing in their state assessments. And, the more we can use essays to assess what students have learned, the greater the likelihood they’ll master important academic content, critical thinking, and effective communication.”

For more than 20 years, companies that provide automated essay scoring software have claimed that their systems can perform as effectively, more affordably and faster than other available methods of essay scoring. The study was the first comprehensive multi-vendor trial to test those claims. The study challenged nine companies that constitute more than ninety-seven percent of the current market of commercial providers of automated essay scoring to compare capabilities. More than 16,000 essays were released from six participating state departments of education, with each set of essays varying in length, type, and grading protocols. The essays were already hand scored according to state standards. The challenge was for companies to approximate established scores by using software.

At a time when the U.S. Department of Education is funding states to design and develop new forms of high-stakes testing, the study introduces important data. Many states are limited to multiple-choice formats, because more sophisticated measures of academic performance cost too much to grade and take too long to process. Forty-five states are already actively overhauling testing standards, and many are considering the use of machine scoring systems.

The study grows from a contest call the Automated Student Assessment Prize, or ASAP, which the Hewlett Foundation is sponsoring to evaluate the current state of automated testing and to encourage further developments in the field.

In addition to looking at commercial vendors, the contest is offering $100,000 in cash prizes in a competition open to anyone to develop new automated essay scoring techniques. The open competition is underway now and scheduled to close on April 30th. The pool of $100,000 will be awarded the best performers. Details of the public competition are available atwww.kaggle.com/c/ASAP-AES . The open competition website includes an active leader board to document prize rules, regularly updated results, and discussion threads between competitors.

The goal of ASAP is to offer a series of impartial competitions in which a fair, open and transparent participation process will allow key participants in the world of education and testing to understand the value of automated student assessment technologies.

ASAP is being conducted with the support of the Partnership for Assessment of Readiness for College and Careers and Smarter Balanced Assessment Consortium, two multi-state consortia funded by the U.S. Department of Education to develop next-generation assessments. ASAP is aligned with the aspirations of the Common Core State Standards and seeks to accelerate assessment innovation to help more students graduate from college and to become career ready.

Jaison Morgan, CEO of The Common Pool, a consulting business that specializes in developing effective incentive models for solving problems, and co-director of the study, said the prize and studies will raise broader awareness of the current capabilities of automated scoring of essays.

“By offering a private demonstration of current capabilities, we can reveal to our state partners what is already commercially available,” Morgan said. “But, by complimenting it with a public competition, we will attract new participants to the field and investment from new players. We believe that the public competition will trigger major breakthroughs.”

ASAP is preparing to introduce a second study, in which private providers and public competitors will be challenged to reveal the capabilities of automated scoring systems for grading short-answer questions. The second study will be conducted this summer. There are another three ASAP studies in development.

I started the JPGB four yeas ago in part out of frustration with the inability of the Manhattan Institute to place and promote my work and in part out of recognition that one no longer needed to go through traditional media outlets to engage in policy discussions.  I figured I could do this myself and on my own terms by blogging.

But I quickly realized I couldn’t do this all by myself.  People advised me that blogs needed regular postings of fresh content, about once every weekday, and I knew that I was not up to writing five posts per week.  So I asked Greg and Matt if they would be willing to post once a week as guest bloggers to ease my burden and keep the blog active and enticing.  Little did I realize how long-running and important their involvement would be.  I feel like they are full partners in this effort and blogging with them has deepened and strengthened our friendship more than I can say.

Very early on it was Greg who proposed the Prime Directive that guides us in this enterprise:  The purpose of the blog is to amuse ourselves.  I started the blog with grand thoughts that it would promote my work and influence policy discussions, but once we adopted the Prime Directive we lowered our ambitions.  How could a crappy little blog change the world?  So we just decided to write whatever we felt like, with no agenda, no inhibitions, and no delusions of influence.

The irony is that perhaps because of our devil-may-care approach, we have probably had more impact on policy discussions than if we were trying to do so.  Our blog posts have spawned news articles, editorials, internet debates, reactions from public officials, and — most importantly from our perspective — a whole lot of fun.

All of this is especially amusing given that all we have is  a domain name, some computers, and a few people devoting their spare time.  Other organizations have paid bloggers working full time, expensive web-designs, and carefully orchestrated PR campaigns and still  can’t gain traction.  Matt, Greg, and I have regular jobs for which we receive no credit or pay for blogging.  We do it because we believe in what we write and enjoy describing the truth as best as we can see it.  The moment this blog becomes affiliated with an organization seeking to advance a particular agenda is the moment it will suck.

Greg and Matt have already done an excellent job of picking some of the best posts from the last four years.  Rather than repeat their good taste, I’d like to use this occasion to describe some of the different types of posts we have on JPGB and illustrate each type with some excellent examples.

Over the last four years we’ve had posts in almost all shapes, sizes, and flavors.  In total we’ve had 1,576 posts, which works out to a little more than one per day.  I’ve written about 642 of them, Matt has written 527, and Greg has 383, with a smattering written by others.  These posts have been viewed more than 718,379 times by readers and have elicited 7,512 comments.  I think I could categorize most of these posts into 7 types:

1) Mocking — I think we are often at our best when we are mocking the sloppy language, sloppy thinking, and herd-like behavior of advocacy groups, bloggers, and journalists.  Some excellent examples of mocking sloppy language include The Fordham Report Drinking Game, Famous Steakholders, Bloggers Shouldn’t Have Rapper Names, Fordham and the Use of Passive Voice, and Buzzword Bingo.  Some excellent examples of mocking sloppy thinking include Hemisphere Fallacy Sighting, Little Ramona’s Gone Hillbilly Nuts, and Gates Foundation Follies (Parts 1 and 2).  And my favorite mocking of faddish herd-behavior includes The Heathers Think-Tanks,  Kevin Carey’s Too Cool for Vouchers… and Cooler Than You, and Valerie Strauss is the Lou Dobbs of Education.

2) Pop Culture — I don’t think I’ve ever seen better and more entertaining analyses of movies, music, and TV than the posts by Matt and Greg on this blog.  In particular, Greg’s marathon examination of the Batman movie, Pass the Popcorn: City of the Dark Knight (Issue #0 through #5) is a masterpiece.  And who could forget his write-up of great summer movies and sequels?  Matt has made his contributions to the movie discussion, notably with his praise of Inglourious Basterds (which I agreed was one of the best movies I’ve ever seen).  But Matt’s bigger pop culture contribution has been in the area of music and kitsch, with posts like Random Pop Culture Apocalypse: Cover Songs,  Random Pop Culture Apocalypse: The Decade in Pop Music and this mashup.  And our pop culture post list couldn’t be complete without mentioning our running commentary on the TV show, Lost.  If only a new series could take its place.  Oh, and then there are lightsabers.  Simply. Awesome.

3) Original Empirical Analyses —  Obviously, a blog is not a place for serious empirical work.  But blogs can feature some interesting facts derived from simple analyses that reveal patterns about the world that were not obvious.  Newspapers used to do this type of simple descriptive work, but now reporters are too busy covering government and Gates Foundation press releases when they aren’t working on their resumes in anticipation of the next round of layoffs.  I did more of these quick and dirty empirical analyses early in the history of the blog and am especially proud of Proximity and Power and Priest and Teacher Sex Scandals in Perspective, which were the two first posts on JPGB.  I was wrote Political Donations from Academia and Arabian Gulf Money and US Universities in the first month of the blog.  More recently I analyzed patterns in school mascot names in a series of Mascot Mania posts.  Matt has also made very very good use of simple charts to illustrate issues we should consider, including on edujobs, alternative certification, teacher quality, and progress in Florida.

4) Recognizing the Unrecognized —  This category can be summed up in two words: Al Copeland.  My favorite winner was Greg’s nomination of Wim Nottroth. But Brian’s nomination of Mary Quant, who did not win, also nicely captures the spirit of The Al.

5) Summarizing Research Findings — We’ve had a number of very useful posts that summarize the research literature, such as this one on participant effects from vouchers, this one and this one on systemic effects, this one on vouchers effects in general, and this one on charter participant effects.  We’ve also highlighted a number of important individual studies, including this one on Head Start, this one on small schools, this one and this one on the Gates Measuring Effective Teachers study, this one on merit pay, and this one on administrative bloat in higher education.  We’ve also notes the foolishness of having a government effort, like the What Works Clearinghouse, attempt to summarize the research literature.

6) Big Think Visions for the Future — Greg and Matt have really excelled in these Big Think pieces.  Matt has a running series on The Way of the Future in American Schooling that describes how digital learning could fundamentally alter (and improve) our system of education.  Matt also has a series of Big Think posts applying Rawls’ ideas of justice to the education system as well as a series advocating Rock Star Pay for excellent teachers (while getting rid of bad teachers and increasing average student teacher ratios).  Greg has some Big Think series that address the philosophical underpinnings of reform strategies, including his series on Command v. Choice and his analysis of incentives and motivation, as well as on the role of science in education policy, such as Vouchers: Evidence and Ideology and The Value-Add Map Is Not the Teaching Territory, But You’ll Still Get Lost without It.   I have my own effort at Big Think pieces, with posts like Build New, Don’t Reform Old, The Dead End of Scientific Progressivism, and Replication, The True Test of Research Quality.

7) Rile Up and Cool Down — This last category consists of two opposite types of posts: those that rile us up against some outrage and those that cool us down to put issues in perspective.  I put them together because they blog really needs both in an appropriate balance.  I tend to get riled up and Matt tends to cool things down (and Greg does some of both, although he tends to do more riling up than cooling down).  For example, I’ve led the blog’s charge against Common Core national standards, lamented the inability of DC folks to generalize beyond their immediate experience, and puzzled over the inability of reporters to accurately summarize research.  Matt, on the other hand, takes a more positive approach, praising the progress that Florida has made and recognizing gains made under Michelle Rhee in DC (while acknowledging the limitations of the  heroic reformer approach).  Greg helps bring balance to The Force by joining the riling up or cooling down side as is necessary.

————————————————————————————-

I want to thank you readers for coming along on this ride.  But I have to tell you that I would be happy to keep blogging even if my only readers were Greg and Matt.  They are the audience I usually imagine when I write a post.  And after posting the first thing I do, quite often, is pick up the phone to ask them, “Did you see what I wrote on the blog?”

It has been an honor blogging with Matt and Greg over the last four years.  And I look forward to keep on doing so as long as the Prime Directive continues to be satisfied.