Jay P. Greene's Blog

(Guest post by Stuart Buck)

Education Week has released its annual report “Quality Counts,” which ranks all fifty states’ education systems along several dimensions, such as school finance, achievement, accountability, and the like.  You can find detailed statistics for any given state on an interactive map, and you can generate a table comparing the states of your choosing.

This Quality Counts report gets a huge amount of attention, as can be seen from the hundreds of results in a search of Google News.

  But the Quality Counts report suffers from two glaring flaws.  In fact, the report reminds me of the old joke (I can’t remember who to credit for this) of a beggar sitting on the streets of New York, with a sign reading, “Wars, 2; Legs Lost, 1; Wives Who Left Me, 2; Children, 3; Lost Jobs, 2.  TOTAL: 10.”  Well, obviously, the number “10” doesn’t represent ten of anything. 

 So what’s wrong with the Quality Counts report?

First, the “School Finance” measure has two basic components: equity and spending.  Equity refers to several measures that look at whether a state’s districts get relatively equal funding.  Fair enough, although there’s a decent argument that impoverished districts might need higher spending to attract better personnel.  But then part of the “School Finance” measure is based on per-pupil spending, as well as the percentage of a state’s taxable resources dedicated to education. 

The problem here is that it doesn’t make sense to reward a state with a higher grade just for spending more, in and of itself.  Indeed, the “spending” measure ends up getting averaged with the measure for “K-12 Achievement.”  This means that, in theory, a state with high spending and low achievement — thus combining incompetence and extravagance — could get an overall score equal to a state with low spending and high achievement.  But if a school manages to get high achievement with low spending, this means that, all else equal, that state has a more efficient and productive education system. 

Second, an even worse problem lies in the “Chance for Success” measure.  This ranking is supposed to tell us about the chances that people in a given state have of succeeding.  There are numerous components to the “Chances for Success” measure, including percent of students above 200% of the poverty line, percent of students with college-educated parents, percent of children whose parents speak English, and more.  Not surprisingly, the richer and more privileged states like Massachusetts, New Jersey, and Connecticut do quite well on this measure, while states like Arkansas, Mississippi, and New Mexico are near the bottom. 

What makes no sense whatsoever is that a high score on the “Chance for Success” measure is averaged together with all the other items — including K-12 Achievement — to produce each state’s final score.  You can see this for yourself: Pick your home state here, and then take the simple average of all six measures (Chances for Success; Standards, Assessment & Accountability; K-12 Achievement; Transitions & Alignment; School Finance; and Teaching Profession), and that average will be the state’s overall final score. 

In other words, imagine a state that managed to produce A-level achievement even though its population was poor and disadvantaged (and thus got a lower grade on the “Chances for Success” measure).  Under any rational grading system, we should give that state the highest possible rating.  But the Quality Counts method would actually downgrade the state for having too many poor children.  By the same token, Quality Counts would upgrade a poor-achieving state that happened to have a privileged and rich student population, even though that state’s education system would obviously be far more incompetent and inefficient.  If anything, the “Chances for Success” ranking should be counted inversely as compared to all the other measures of a state’s education system. 

This entry was posted on Friday, January 9th, 2009 at 5:00 pm and is filed under school accountability. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.