I’ve lost track of how many studies with rigorous causal identification find that improving test scores is not associated with improving later life outcomes, like graduating high school, attending and completing college, and getting a job and earning higher salaries. In the area of school choice alone we now have dozens of such studies, but you can also find this achievement-attainment disconnect in rigorous studies of pre-school and other interventions.
You can add to this list the latest study from Mathematica on the long-term effects of attending a charter middle school. Mathematica examined lotteries at a sample of charter middle schools around the nation, comparing outcomes for lottery winners to those of lottery losers. When the original results were completed almost a decade a ago, they found no overall effect on test scores. But when they disaggregated results, they found that urban charters had a significantly positive effect while suburban charters had the opposite result. This prompted leaders in the charter and ed reform worlds to conclude that we should focus our efforts on urban charters, since those were the ones that “worked.”
Now Mathematica has followed-up on those students to see whether they eventually attended and completed college. The new results show that attending a charter middle school has no effect on attending or completing college, just as it had no overall effect on test scores. But — and this is the important part for our discussion — they also concluded:
The success of an individual charter middle school in improving college outcomes was not related to its success in improving middle school achievement. The study schools that improved middle school achievement were not consistently more successful than others in boosting college enrollment and completion.
So, all of those technocratically-minded ed reformers who thought we should focus on urban charters because test scores showed they “worked” were guilty of mis-judging long-term success based on unreliable short-term measures. The evidence shows that changing test scores is not a particularly good indicator of schools that will improve their students’ lives.
City Fund, NACSA, and others who support portfolio management, harbor-mastering, quarterbacking, or whatever marketing term they are using nowadays are once again left trying to explain exactly how they intend to distinguish the “good” charter schools from the “bad” ones better than parents can. But take comfort, their political ineptitude matches their technocratic inclinations so that I expect they will burn through their $200 million without successfully installing and maintaining any multi-sector portfolio management systems. So children will be safe from their falsely-guided superior judgement.
Jay P. Greene's Blog 
“The evidence shows that changing test scores is not a particularly good indicator of schools that will improve their students’ lives.”
OK–what are better indicators for, say, parents of fifth graders going on to middle school? Can those parents ask a charter middle-school principal how that school has improved its students lives?
Only technocrats require a metric for decisions and will insist on maintaining a flawed one unless a superior one can replace it. Most people just exercise human judgement, as flawed as that may be. How do people pick the right piano teacher, or decide which sport their kids will play, or select a babysitter? Even for very important decisions, like who should I marry, people rightly do not seek faux-scientific metrics.
They just do their best exercising judgement.
Perhaps you are right but, then, what do the increased divorce rates speak of the quality of subjective selection of marrying partners? At least in the past it was more mechanical (even if possibly imperfect) criteria: wealth, family background, common cultural heritage (smile). Not to mention the number of abandoned “friends” over the years or fired nannies and piano teachers.
What’s your control group for that analysis?
At the high school level you can at least look at graduation and college enrollment rates, which I believe are meaningful for long term outcomes. But subjective parental judgment of qualitative factors is likely to be more valuable even then.
Well, yes. But how much are those influenced by the same — possibly meaningless — HS scores? After all, you can’t enroll in college if you don’t have high enough HS scores, and you can’t graduate college if you don’t enroll. So all of those may possibly be vicariously correlated with HS scores, and colleges indeed reflect signalling more than anything else (sad smile).
HS graduation and college enrollment are connected to life outcomes while test scores are not. You’re grasping at straws.
Our (admittedly non-scientific) 2018 survey of 1,500 Michigan charter school parents included a 5-point Likert scale question to rate the importance of 20 different factors to them in exercising choice. Not a lot stood out, but we did find: 1) a greater intensity among lower-income and less educated parents for choosing all factors; and 2) on average, standardized test scores mattered LESS than safety, teacher/principal relationships, technology and small class sizes but MORE than diverse student body, sports, afterschool programs and access to transportation.
Technocrats tend to put too much emphasis on standardized test scores, but especially for some demographics of parents (at least in Michigan) they do matter to some extent. https://www.mackinac.org/archives/2018/SchoolChoiceSurvey2018-web.pdf
They can visit the school, talk to parents who’ve enrolled their children, talk to former students. I’ve said on this blog a few times if you judged me by my test scores I kinda suck; if you talk to former students and parents the vast majority point to me as the foundation of their child’s later academic success, in math particularly.
Cateaux! I rescind ze ordeur for yew to attack!
After more than 25 years of living in this milieu, I am close to concluding that:
– One can believe that excellent education leads to creation of some fraction of excellent people. A couple of thousand years of history may point this way, although many counter examples exist too.
– No amount of education research (even without quotation marks!) will be ever there to guide us what is a “better” or “worse” education, in the broad sense of the word. Any measure of broadly accepted educational “excellence” has been long corrupted by colleges caving to the social pressures of being seen as “equitable” and “diverse,” workplace yielding to similar (plus legal) pressures of equitable and diverse hiring, and internet and social media promoting a byzantine collection of ever-changing values as criteria for success and “excellence,” makes such pursuit of broadly acceptable measure meaningless. To make it even worse, even such un-corrupted measures wouldn’t do anymore, as most of us believe today that each individual is entitled to define excellence in a personalized way.
So we are, indeed, in an uncharted territory, and no research will show us the way.
If test scores don’t work, it doesn’t mean nothing does. See for example the emerging line of research on character formation.
We go back to the old discussion about the purpose of *public* education. Is it a workplace development? Character development, however defined? Indoctrination to create conformist public? Intellectual/scientific development of the individual? Something else (I can think of a few more)?
When we had a strong dominant line of public sentiment, character development (educated citizenry to preserve the Republic) seemed reasonable. No more. With the advent of science, intellectual/scientific development took over a bit at certain times (e.g., late 19th c., Sputnik era). Progressives pushed workplace development and conformity at different times. But we seem way beyond that today. So where are we?
It’s not hard to look into the research and discover that you are not the first to ask these questions, and good answers are less impossible than you may expect.
Good post. I was surprised that only 10% of the 31 schools were high poverty schools (75%+ free/reduced lunch). So only three such schools in the whole study? I may be misreading this.
If true, perhaps that would limit some of the claims of what this particular study suggests vis-à-vis City Fund, as I think they mostly fund high poverty schools.
And if I can proffer an olive branch for you and them, I think their main expenditure remains simply giving money to popular schools to expand, because fortunately there is a correlation in most cities between popular schools and test score gaining schools. Perhaps you can reject the technocratic part and still cheer for more parents getting seats in schools they want.
Anyway, your broader point about caution with test score gains still applies.
As I read it, I think Jay only does reject the technocratic part.
Jay, what if there’s more to college-going than any K-12 school can quite address?
E.g., finances, family situations, pressure from peers and neighborhoods.
If a KIPP graduate thinks, “I wish I could go to college, but my single mom really needs me to help earn some money and take care of my 12-year-old sister,” does that mean that KIPP isn’t an effective intervention? And if your logic is taken far enough, is there anything that would ever be an effective intervention for anything? In the long run, we all die.
Actually, last time Jay and I looked, the number of graduating HS students academically qualified to attend college each spring and the number of new students enrolling in college each fall were close to identical. There just isn’t a large population of kids who even *could* fit your description here. Whether sending more kids to college should be the goal is open to debate, but if that’s the goal, K12 is the place to solve it.
The best you can ever hope for from any intervention (from eyeglasses to capitalism), is that it improves one aspect of life for at least a few people for a reasonable amount of time. Nothing is permanent, nothing improves all of life. Ultimately it’s a bit of a cheap shot to find some downstream outcome that isn’t affected. Even the best interventions fade or dissipate entirely. An infection at age 40 isn’t helped by the fact that I once took penicillin at age 25. So what?
But some interventions do have lasting effects. Eyeglasses don’t, but corrective eye surgery does. And anyway, the question here is not “how long do improvements last?” but “what counts as an improvement?” If test scores don’t correspond to long term outcomes, why do we care about them at all?
We have some measures (HS graduation, college enrollment) that do correspond to long term effects. If school choice raises test scores only a little but raises HS graduation a lot, then school choice would seem to have a lot of long-term positive value for students who use it. Other interventions that raise test scores more but don’t budge grad rates don’t seem to have much value.
Two points: Graduation rates are not a meaningful sign of success if standards are ever watered down (otherwise, it would be trivial for schools to raise the graduation rate to 100% by automatically conferring a degree on everyone no matter whether they attended or passed any classes). Conversely, the positive value of graduation has a lot to do with signaling in the marketplace, not actual educational value.
Put those two points together: I’m wary of the idea that schools should deemphasize tests (which at least could measure whether the kids have actually learned some math or science), and instead push everyone to somehow skate through graduation even if that waters down the standards (and ultimately the signaling benefit).
Suppose we don’t *force* schools to do anything, but also don’t spread the idea that nothing is effective in the long term when it looks like some things are? It’s just crazy enough to work.
[…] that we hope translate into greater educational and life success for students. Unfortunately, a growing body of research is showing a disconnect between changing test scores and changing later life […]