(Guest Post from Matthew Ladner)
We now have a bipartisan analysis of what actually happened in Indiana grading-gate scandal. Rick Hess covers the subject here:
Flash-forward five weeks, and we finally have a resolution. The headline: Bennett exonerated. That’s the conclusion of a 56-page official report, requested by Indiana’s legislative leaders, and released Friday. Authored by Democrat John Grew, executive director of state relations and policy analysis at Indiana University, and Republican Bill Sheldrake, president and founder of Indianapolis-based research firm Policy Analytics, the report finds that Bennett acted appropriately and fair-mindedly. Grew and Sheldrake spent the past month or so investigating what happened and reviewing the data. They concluded that Bennett and his staff made “fair” and “plausible” changes to Indiana’s school rating system before releasing 2012′s A-F grades. They found that a lack of planning and capacity had forced Bennett and his team to make a series of on-the-fly “interpretations” and judgments, but that Bennett and his staff “consistently” applied changes to Christel House and 180 other affected schools. In short, nothing to see here.
Since Tony’s critics are on the whole fair-minded people with only a tiny minority suffering from some sort of derangement syndrome, I’m sure Tony’s inbox will be filling up with apologies. Some analysts who were willing to pontificate much with little in the way of facts just might be feeling a bit sheepish today as well. Pundits are extremely responsible after all and never just shuffle on to the next subject when they are way off base on something.
I have believed from the outset that no one from Diane Ravitch to Charles Murray sitting in Tony’s position would have told 16 schools without junior and senior students to simply eat getting zero points from graduation rates and AP completion categories. “Would you shut up already and get some juniors and seniors” is simply not a response that any half-way reasonable person was going to utter. I have also believed from the outset that if the changes made applied to only one school then it was a scandal, but that if they evenly applied the changes across schools then this was a hatchet-job.
For the record it was a hatchet-job.
Jay P. Greene's Blog 
Nicely-done!
I for one apologize for nothing. All I did was ask a question about what happened. I repeatedly stated that I’d be happy to hear whatever was the true answer, but it was a question that I felt (and still feel) really needed to be asked.
And here’s the thing. Nobody was asking. Some people seemed very anxious that the question not be asked. In such an environment, asking the question is responsible. I apologize for nothing.
Your statement that there were only two possibilities – “scandal” and “hatchet job” – seems to leave me with no place to stand. There was enough ambiguity in the initial reports – and in Bennett’s responses – to justify media outlets running the story and readers like myself asking Bennett to explain himself. I’m glad the story was reported and the hard questions got asked.
Or do we want to send the signal “if you are identified as part of the education reform tribe, any time it looks like there might be a question about your behavior we will rush out to defend you and condemn as evil anyone who asks questions”? Count me out.
I don’t read this as the exoneration that you so desperately want to see – the point that I would make is that Bennett and his cronies went into overdrive to salvage Christel House’s reputation — they had a pre-conceived notion that the school would fare better. But they also had a pre-conceived notion that the public schools that operate in urban settings would fare worse — yet they turned a deaf ear to any concerns that emanated from the public school community……I won’t defend the IPS and Gary school districts of the world as I think they are terrible and need competition — but they deserved better than they got from the Bennett regime. Bennett may have acted “responsibly” in changing the grades for schools other than Christel House, but you won’t convince me that they acted in the interests of the law, the a-f grading policy or more importantly, the larger school reform movement. This was a setback.
What would you have done if you had discovered a dozen plus schools without junior and seniors and found yourself with a formula that judged high schools on the basis of graduation rates and AP exams?
Here’s how Slate magazine interpreted what Bennett and his cronies did: — they can say it better than I.
“Bennett’s explanation is perfectly mathematically reasonable, and it would get him off the hook. The only problem is that the story he’s telling appears to be totally false.
Bennett told AEI’s Rick Hess, “As we were looking at the grades we were giving our schools, we realized that state law created an unfair penalty for schools that didn’t have 11th and 12th grades. Statewide, there were 13 schools in question had unusual grade configurations. The data for grades 11 and 12 came in as zero. When we caught it, we fixed it.”
Bennett’s stated rationale makes sense. Here’s an analogy. You’re teaching a course with three exams, and each student’s overall exam grade for the class is computed as an average of her three individual exams
(1/3) * (exam 1) + (1/3) * (exam 2) + (1/3) * (exam 3)
But what happens if a student misses exam 3, with a justified absence, and the test can’t be made up? Then we have what’s called a “missing data” problem—we have to infer something about the student’s performance without the full complement of data we have for everybody else. One natural approach is to compute that student’s overall exam grade as the average of the exams she did take:
(1/2) * (exam 1) + (1/2) * (exam 2).
What you shouldn’t do is give the student a zero for the exam and average it into her grade.
Christel House had started out as a middle school and was adding one high school grade each year; for the 2011-12 school year, it served students through the 10th grade. High schools, according to Indiana statute, were to be graded on four metrics, averaged like so:
30% * (English scores) + 30% * (algebra scores) + 30% * (graduation rate) + 10% * (college and career readiness score)
But, as Bennett said in an official statement, “Christel House only served students in grades K-10, thus the graduation rate and college and career readiness measures could not be calculated because the school did not serve grades 11 and 12.” So it faced a missing data problem, like the student who had to miss an exam.
Indeed, it wouldn’t be fair to count those scores as zero! The obvious fix, just as with the student, is to grade such a school on the two scores that it does have:
50% * (English scores) + 50% * (Algebra scores)
Does that mean Bennett really did get railroaded, and he was just fixing an obvious error?
No—because the “fixed” version of the grade was what Indiana was already using before Bennett started tinkering with the gears. When you dig into the numbers, the story about the unfair zeroes looks like a complete fabrication.
Christel’s ninth- and 10th-grade students got brutally low scores on the English and math tests, with only 70 percent passing English and just a third passing math. Here’s how Jon Gubera, then the Education Department’s chief accountability officer, described Christel’s performance in one of the emails released by the AP:
“OK, here is their breakdown …
They served grades K-10 in 2011-12 so they are a combined school but do not have any graduates. So their grade is a combination of the standard E/MS model and the HS 9&10 model which only counts ECA proficiency.
E/MS results: 3.00 on E/LA (no growth bonuses) and 4.00 on math (bottom 25% bonus) = 3.50 points (B)
HS results: 2.00 on E/LA (70% pass rate) and 0.00 on math (33% pass rate) = 1.00 points (D)
Final Combined results: E/MS 3.50 x .76 (76% of school is in grades 3.8) = 2.66 + HS 1.00 x .24 (24% of school is in high school) = .24. Thus overall grade is 2.66 + .24 = 2.9 (C).
Bottom line: their terrible 10th grade Algebra I results (33% passing) was the principal factor in earning a C grade.”
Christel’s grade 3–8 scores came to a 3.50, or a B, and their high school scores, thanks to the algebra fiasco, were a 1.00, or a D. The school’s score is then an average of the grade 3–8 scores and the 9–10 scores, weighted according to the proportion of students in each group.
Where are the zeroes for the graduation rates and readiness score that got averaged into Christel’s score—the “unfair penalty” that Bennett claims he fixed? They’re not there, because that’s the part that Bennett, as far as I can tell, simply made up.
I don’t pretend to understand the arcana of Indiana’s 2012 formula, but the report states:
“Because the Bennett administrative rule did not contemplate all of the numerous school configurations in place during 2011-12, it was necessary for IDOE to make certain interpretations including
the decision to eliminate HS. scores from the Christel House Academy’s grade. The authors found that this interpretation was consistently applied to 16 other schools which had analogous
situations.”
Later it says “In the end, the Authors found that the two adjustments administered to determine Christel House Academy’s final grade were plausible and the treatment afforded to the school was consistently applied to other schools with similar circumstances.”
The questions were asked but too few people waited for the answers. If you want to see some examples, click the link in the text of the post.
Yes, you and I can definitely shake hands and agree wholeheartedly that Kevin Carey is a colossal asshole. But then, that’s just another way of saying today is a day of the week that ends in “y.”
Isn’t it nice to know that whatever our disagreements may be, there are some fundamental cosmic truths we can always come back to when we seek the unity of the human spirit?
I actually find Carey’s higher ed work entertaining. I went back to read those pieces thinking that perhaps I would be able to say that his didn’t top the charts for premature sanctimony.
As it turns out, I can’t.
A question for the philosophers: is Kevin Carey’s sanctimony “premature” if it is uninformed by the evidence not because he didn’t wait to learn the facts, but because he doesn’t care about them?
It was premature sanctimony only because he jumped to conclusions without waiting for the facts. Now that we have a good read on what happened, it is both premature and inappropriate.
In the end though Carey was just one of the animals in a stampede. No one should be so easily fooled by a collection of out of context emails and an obviously politically motivated story. Elements of the press and commetariat fell for this hook, line and sinker. As Hess said, what office does Bennett visit to get his reputation back?
The biggest hit to Bennett’s reputation was caused by his own resignation.
It was only a handful of Florida Dems and the usual blogosphere suspects asking for him to leave. The claim that he would have somehow embarrassed/hamstrung the governor or SED every time he opened his mouth strikes me as being borderline laughable.
Maybe even with this report (NB: is this a new world speed record for a bipartisan investigative committee? nice job, Indiana!), we still don’t know the full story. And I don’t say this to imply there were misdeeds to the committee overlooked: maybe he simply wanted an “out” from Florida, maybe he was pushed into it, maybe he just panicked and overreacted. The PR takeaway here is that if you haven’t done anything wrong and you want to protect your reputation, resigning probably isn’t the way to go.
The emails the AP published were the indictment of what Bennett and crew did. In their own words.
Charlotte-
If you feel that way please feel free to email me your entire Outlook file for the last six months. I post them in a google doc, and we can have a contest to see who can develop the best conspiracy theory about you with any four emails taken completely out of context. It will be fun for all involved!
Matt:
I appreciate your loyalty to Tony, a superintendent who did some great things, some somewhat ok things, and some boneheaded things.
From start to finish (comments in articles, interviews on NPR and all the rest), we have been consistent in saying these three things:
1) accountability systems are not to prove out that what we see with our bare eyes as great schools are great schools. Case in point, Christel House is by all reports by those familiar with the school raising student achievement levels. I am not familiar enough with that specific school to know, but when I see only 33% of 10th graders (yes, 10th graders) in Algebra I performing at a proficient level, I think Gubera was right to write that it was tough to categorize that an A level school. Remember: That’s Algebra I. In MA, we do Algebra I in 8th grade. (Yep, Common Core in 9th and 10th grade.)
2) That Tony wd need to change the A-F grades through a systematic redefinition is not unheard of when you are dealing with the complex bureaucratically defined school grading systems. Mike Petrilli called his changes a “calibration”… Ok, I accepted and continue to accept that at face value. The problem is that as a state official reporting out public grades for public schools, he needed to make public (and really get approval from an objective body… Board of Ed?) to do that. I would hope the initial definition of the grading system was approved by the Board and not just concocted by 3-4 people in a state ed office. Accountability systems are only worthy of the public trust if you make sure changes made are public. Otherwise, who knows what goes into the black box?
3) the A-F grading system is not the way to go. Better for the state ed dept to provide student test scores and other student performance data to the public. Newspapers, magazines, etc. (intermediary associations of civil society, not government) can create their own easy-to-understand grading. That’s how it works in MA, and also for cars or any consumer product. We’ve had none of the heartache and bureaucratic fixes IN, FL and other states have seen. Why create hard political pressures (or the perception of them) that lead to idiocies like not allowing grades to rise or fall more than a single grade (FL) or closed-room redefinition s of accountability (IN)? It is hurting out case of reform, entrusts bureaucrats with too much defining information, and mistrusts the public’s and civil society’s ability to navigate information.