In my final piece of this series arguing against the high-regulation approach to school choice, I’m going to discuss whether achievement test results are a reasonable proxy for school quality. Achievement tests are at the center of the high-regulation approach. They are used by regulators — whether authorizers, portfolio managers, or harbor masters — to identify good and bad schools, to determine whether they should be included as choice options, and to shape the goals schools should pursue.
There is no question that growth in student learning provides us with some useful information. The problem is that school quality is much broader than just test score results. I always understood that achievement tests were only a partial and imperfect indicator of school quality, but I used to believe that other aspects of school quality not captured by achievement tests were largely correlated with those test results. That is, I used to think that if a school raised scores it probably meant that students were safer, more students would graduate, more students would learn productive values, and more students would go on to become successful adults.
Unfortunately, the evidence is increasingly clear that test scores are only weakly correlated with all of these other desirable outcomes from schools. All you have to do is look at yesterday’s post. Schools that produce the largest achievement test gains are not necessarily the ones that produce higher graduation, or college-attendance rates. And sometimes schools with unimpressive achievement gains make significant contributions to attainment and annual earnings when students join the workforce. I used to think that this couldn’t be possible. All of these happy outcomes had to be aligned. They just aren’t.
If you are not persuaded by the evidence I reviewed yesterday on the disconnect between achievement results and other outcomes, I suggest you read an excellent book written by Nobel Prize winning economist James Heckman and his students called The Myth of Achievement Tests.
The problem is that the high-regulation approach needs achievement tests to be correlated with all of these other good outcomes. If they are going to pick the school choice winners and losers based on test scores, then test scores need to be strongly predictive of other things we care about. People have been very slow to accept the fact that test scores are only weakly correlated with later life outcomes because it would be so convenient if readily available and relatively inexpensive test scores could capture something as complex as school quality. The fact that they don’t throws a monkey wrench into the entire high-regulation machinery.
The reality is that the average low-income parent has more complete information about their kid’s school quality than does the highly-trained regulator armed only with test scores. When we wonder why parents are choosing schools that regulators and other distant experts deem to be “bad,” it is almost certainly because the parents know more about what is good and bad than do the experts.
The wrong response to recognizing that test scores fail to capture school quality sufficiently is to increase the set of high-stakes measures we collect. We can’t fix the limits of math and reading achievement tests by adding mandatory “grit” surveys or other measures. Even informed by a variety of measures, Chinese officials are no more effective in telling state-controlled banks how to allocate capital than portfolio managers are in determining how to allocate school options. Decentralized decision-making is simply better than central planning.
The school choice movement has to remember that choice is what makes this reform work, not the regulation. I’m perfectly willing to accept that some regulation is necessary and inevitable. And I’m willing to make compromises to get programs adopted. But the cardinal sin of the high-regulation school choice folks is that they believe that heavy regulation is the ideal and should be the starting point for political compromises.
Jay P. Greene's Blog 
https://qph.is.quoracdn.net/main-qimg-f60cae383b688d46b6f6424ab4c35b70?convert_to_webp=true
Brian, that cartoon is a good reminder that the historical association of technocracy with “social Darwinism” is not an accident. We shouldn’t fool ourselves that *this* time when we take away people’s control over their own lives, we won’t become arrogant racist paternalists like we did last time.
I’ll repeat myself whenever this comes up:
Our school’s test scores are above-the-neighborhood but below the state ‘standard’. Parents choose our school because their children are save, loved, and when they leave us they graduate from HS and go onto college.
I’ve gotten two consistent forms of feedback as an 8th grade teacher; either “Thank you; math is easy because of your class,” or ” I wish I paid attention in your class; we’re doing the same thing!”
Our charter school would not be surviving without the simple feedback we get from parents: they enroll their children into our school, and stay.
Hi Jay,
Yesterday you wrote: “For reasons I will discuss at greater length in the next post, I think attainment is a more meaningful indicator of long-term benefits than achievement test results.”
1. I may have missed it — but were you going to make the affirmative case for attainment in this post?
2. My view –
I.e., I agree — test scores don’t mean much unless they correlate to something “real” later. Labor market outcomes, for sure. College success, probably. High school graduation from really easy urban high schools, I’m less sure.
3. What’s the empirical test which can answer which is the better indicator?
If, for example, Angrist et al produce a study of labor market outcomes 10 years out….and the charter kids with high test gains turn out to have higher incomes, and the control group with higher 4-year graduation is lower, will you switch your view on which is the better indicator of long-term benefit?
Hi Mike, Sorry I ran out of steam and did not elaborate on that point as I planned to. Basically, I agree with you that long-term outcomes are more important. And the connection between schools increasing achievement scores and increasing later life outcomes is weak. I’m basing that conclusion of the weak connection on my review of those 7 charter and private school choice studies as well as the Heckman book I referenced. If Angrist is finding a stronger connection that would be encouraging, but it has to be set within a larger literature that does not find gains in achievement correlated strongly with gains in attainment and earnings. Of course, there is also the Chetty and Rockoff study, but the effect is not huge.
A 12.7% increase in pay may or may not be “huge” but it is very meaningful in my book.
Thanks for your response.
One small nugget vis a vis the Angrist study. If you saw what Boston district high schools are like to graduate — the no-homework, no classwork, sheer nothingness; versus charters that at least pass what I’m sure you’d think was a reasonable bar to have HS graduation — I wonder if it might change a bit how you view high school completion as a valuable proxy for anything.
That’s a good point, but college-going rates are not under school control and cannot be explained by the application of different standards in charter and traditional public schools.
Hey Matt — I’m not sure where you get 12.7% from. Chetty, Freidman, and Rockoff found about a $200 increase in earnings from a one standard deviation improvement in teacher value-added on achievement tests. Meanwhile, Booker, et al found over a $2,000 increase in annual earnings from attending a charter school in Florida even though those charter schools produced little improvement in achievement scores.
Ooops got my list of last names mixed up.
Really interesting post. One related piece that is on the top of my mind having just looked for a school for my kindergartener is this – amongst middle and upper middle class parents looking for public schools, test scores are hugely important drivers. I have friends that won’t consider anything less than a 9 or 10 on GreatSchools. If parents actually aren’t great evaluators of school quality either, where does that leave us?
I’m not sure all parents rely as much on achievement results as do your friends, but you make a good point. Parents will make mistakes. Sometimes they will rely too much on a weak indicator or give too much priority to superficial factors, like new buildings. My point is simply that, on average, parents have more complete and better information and will tend to make better decisions than will distant regulators armed only with test scores. We will never live in a perfect world, but I think we’ll get fewer mistakes if we rely more on parents and less on regulators. And if mistakes are made, I’d rather that families make them for themselves rather than a bureaucrat making it for them. All this being said, I recognize that there is some role for regulation and regulators. I’m just saying that the high-regulation folks have gone way too far in having confidence in their ability to distinguish good from bad schools.
We are in total agreement on the limits of achievement tests. However, there’s plenty of evidence that shopping for schools is complicated by information asymmetries between schools and parents and extensive stratification in access to information between low-income and more affluent parents. These challenges may be addressed by low-cost interventions that improve access to information. But even if we improve access to information, well known cognitive biases are likely to result in many mistakes. Like you, I’m okay with people making bad choices except when these choices have negative externalities that affect society as a whole. It may be that government will be more effective at establishing performance floors – much like they do in the arena of auto safety – rather than driving continuous improvement in school quality.
This sounds like a reasonable approach to me. The government could play a role in improving consumer information about schools, but we have to be careful that we don’t force schools to attend to a single measure that fails to reflect quality adequately. And I agree that regulators can play a better role trying to establish a floor rather than drive continuous improvement.
I suspected we weren’t that far apart in our thinking even though we tend to emphasize different things.
Okay, I don’t necessarily agree with you guys ideologically. I do appreciate the post above greatly. I have a couple of questions though.
The biggest driver of creating school choice, and charter schools in particular, were the achievement rates provided on test scores through NCLB. I am aware that your group has believed in school choice preceding NCLB but with those test scores, your school choice scenario is highly unlikely. The low test scores were the impetus to lift charter caps.
I’m not going to engage in some debate of charter schools because there’s evidence and research to support pretty much any philosophical inclination. But you do realize that charters have leveraged their test scores into their proof of superiority. This has led, in my opinion, to lots of gaming of education models and student populations for the purpose of higher test scores, or at least improving test scores, as the Holy Grail. I teach in high school and I adjunct at a local community college. Charter grads in my college class have frequently told me about the high volume of test prep they absorbed while in high school. Multiple students told me that had an ACT prep course for their entire junior years, which is a luxury that traditional public schools could never pursue. (I guess your choice argument is valid here since charter parents must be okay with a test prep education while my students’ parents would consider such pursuits to be distasteful.)
So I guess my point is that test scores are fetishized. They’ve become THE goal of schools of any sort. If that’s the case, then test scores should be better than ten years ago when the tests had less importance. I mean, if tests are used as part of teacher evaluation, doesn’t it behoove the teachers to gear things for the tests. The very tests that you just said are not truly a marker of quality.
When I listen to some charter network leaders talk about their models, they often openly state that they spend disproportionate amounts of the school day on tested subjects (English and math). They’re pretty open about it taking 80% or more of their school day. I’ve read your appreciation of the arts and museums in education. Doesn’t that model conflict with some of your core values? But you gotta admit, it sure improves those test scores!
What I think we’re seeing is a consistent gaming or strategizing for test scores and confusing that for educational achievement.
Lastly, in reading many of the posts that follow this one, I see that you justify your policy prescriptions with test scores often. If the scores are not an indicator of quality, then why are they referenced as meaningful for policy?
Jay,
I’ve put the Heckman book on my to-read list, but my mental library cites Charles Murray’s work on IQ and such when discussing the accuracy of math and reading tests, as well as E.D. Hirsch’s work. Am I not remembering correctly that both say math and verbal scores (Hirsch says especially verbal, of course) correlate with things like marital happiness, income, and leadership positions? Murray cites a thing he labels G in Real Education, and notes that verbal ability correlates even with physical/sports ability (although not as highly as it correlates with other things, like income).
If that is the case, am I wrong to think that these test scores do indeed tell us a lot with one simple test administration? If not perfectly, at least a pretty darn good general approximation for a simple reading/vocab test? If Heckman answers these objections, just lemme know and I’ll go read the book instead of asking for your reply.
Joy
The question is whether a change in achievement causes a change in long-term outcomes. After all this is why people focus on test scores. They believe that raising test scores will improve kids lives.
Many have found that the level of achievement is associated with later outcomes, but that is not causal. Chetty, Friedman, and Rockoff find that a school induced change in achievement appears to be causally related to increased earnings but the effect is modest and there are some questions about whether we are confident that it is causal.