Quality Counts Lacks Quality

(Guest post by Stuart Buck)

Education Week has released its annual report “Quality Counts,” which ranks all fifty states’ education systems along several dimensions, such as school finance, achievement, accountability, and the like.  You can find detailed statistics for any given state on an interactive map, and you can generate a table comparing the states of your choosing.

This Quality Counts report gets a huge amount of attention, as can be seen from the hundreds of results in a search of Google News.

  But the Quality Counts report suffers from two glaring flaws.  In fact, the report reminds me of the old joke (I can’t remember who to credit for this) of a beggar sitting on the streets of New York, with a sign reading, “Wars, 2; Legs Lost, 1; Wives Who Left Me, 2; Children, 3; Lost Jobs, 2.  TOTAL: 10.”  Well, obviously, the number “10” doesn’t represent ten of anything

 So what’s wrong with the Quality Counts report?

First, the “School Finance” measure has two basic components: equity and spending.  Equity refers to several measures that look at whether a state’s districts get relatively equal funding.  Fair enough, although there’s a decent argument that impoverished districts might need higher spending to attract better personnel.  But then part of the “School Finance” measure is based on per-pupil spending, as well as the percentage of a state’s taxable resources dedicated to education. 

The problem here is that it doesn’t make sense to reward a state with a higher grade just for spending more, in and of itself.  Indeed, the “spending” measure ends up getting averaged with the measure for “K-12 Achievement.”  This means that, in theory, a state with high spending and low achievement — thus combining incompetence and extravagance — could get an overall score equal to a state with low spending and high achievement.  But if a school manages to get high achievement with low spending, this means that, all else equal, that state has a more efficient and productive education system. 

Second, an even worse problem lies in the “Chance for Success” measure.  This ranking is supposed to tell us about the chances that people in a given state have of succeeding.  There are numerous components to the “Chances for Success” measure, including percent of students above 200% of the poverty line, percent of students with college-educated parents, percent of children whose parents speak English, and more.  Not surprisingly, the richer and more privileged states like Massachusetts, New Jersey, and Connecticut do quite well on this measure, while states like Arkansas, Mississippi, and New Mexico are near the bottom. 

What makes no sense whatsoever is that a high score on the “Chance for Success” measure is averaged together with all the other items — including K-12 Achievement — to produce each state’s final score.  You can see this for yourself: Pick your home state here, and then take the simple average of all six measures (Chances for Success; Standards, Assessment & Accountability; K-12 Achievement; Transitions & Alignment; School Finance; and Teaching Profession), and that average will be the state’s overall final score. 

In other words, imagine a state that managed to produce A-level achievement even though its population was poor and disadvantaged (and thus got a lower grade on the “Chances for Success” measure).  Under any rational grading system, we should give that state the highest possible rating.  But the Quality Counts method would actually downgrade the state for having too many poor children.  By the same token, Quality Counts would upgrade a poor-achieving state that happened to have a privileged and rich student population, even though that state’s education system would obviously be far more incompetent and inefficient.  If anything, the “Chances for Success” ranking should be counted inversely as compared to all the other measures of a state’s education system. 

6 Responses to Quality Counts Lacks Quality

  1. Lionthrax says:

    Mr. Buck hits a homerun with this analysis! I jumped out of my chair cheering the fact that someone had the audacity of explaining how the ‘experts’ are so fatal flawed in their view of education.

    The only way to explain the contrast between what is and how it is percieved by the educrats is that they have no real concept of the fundamentals of education or in essence how to educate individuals.

    The education system is all about $$, and it has been proven that money does not alter the quality of education.

    Again, kudos to Mr. Buck!

  2. Roger Sweeny says:

    The beggar story reminded me of a this passage from physicist Richard Feynman, part of his experience reading math texts for a California adoption commission in 1964:

    “Finally I come to a book that says, “Mathematics is used in science in many ways. We will give you an example from astronomy, which is the science of stars.” I turn the page, and it says, “Red stars have a temperature of four thousand degrees, yellow stars have a temperature of five thousand degrees . . .” — so far, so good. It continues: “Green stars have a temperature of seven thousand degrees, blue stars have a temperature of ten thousand degrees, and violet stars have a temperature of . . . (some big number).” There are no green or violet stars, but the figures for the others are roughly correct. It’s vaguely right ….

    “Anyway, I’m happy with this book, because it’s the first example of applying arithmetic to science. I’m a bit unhappy when I read about the stars’ temperatures, but I’m not very unhappy because it’s more or less right — it’s just an example of error. Then comes the list of problems. It says, “John and his father go out to look at the stars. John sees two blue stars and a red star. His father sees a green star, a violet star, and two yellow stars. What is the total temperature of the stars seen by John and his father?” — and I would explode in horror.”


  3. […] do I say that? Check out legal scholar and philosopher Stuart Buck’s quick work of deconstructing Quality Counts. First, the humorous: In fact, the report reminds me of the old joke (I can’t remember who to […]

  4. […] critique of the Quality Counts Report over at Jay Greene’s blog last year. You can see it here. But for those of you who are lazy here is a little […]

  5. […]  Graduate student Stuart Buck wrote a thoughtful blog post criticizing the ranking system here, and Stuart and I together penned a Letter to the Editor published in the February 4, 2009 issue […]

  6. […] addition, here’s a good explanation of the shortfalls of the Quality Counts survey from a couple years b… by all-around smart guy Stuart Buck from the University of Arkansas Dept. of Education […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: