I’ve lost track of how many studies with rigorous causal identification find that improving test scores is not associated with improving later life outcomes, like graduating high school, attending and completing college, and getting a job and earning higher salaries. In the area of school choice alone we now have dozens of such studies, but you can also find this achievement-attainment disconnect in rigorous studies of pre-school and other interventions.
You can add to this list the latest study from Mathematica on the long-term effects of attending a charter middle school. Mathematica examined lotteries at a sample of charter middle schools around the nation, comparing outcomes for lottery winners to those of lottery losers. When the original results were completed almost a decade a ago, they found no overall effect on test scores. But when they disaggregated results, they found that urban charters had a significantly positive effect while suburban charters had the opposite result. This prompted leaders in the charter and ed reform worlds to conclude that we should focus our efforts on urban charters, since those were the ones that “worked.”
Now Mathematica has followed-up on those students to see whether they eventually attended and completed college. The new results show that attending a charter middle school has no effect on attending or completing college, just as it had no overall effect on test scores. But — and this is the important part for our discussion — they also concluded:
The success of an individual charter middle school in improving college outcomes was not related to its success in improving middle school achievement. The study schools that improved middle school achievement were not consistently more successful than others in boosting college enrollment and completion.
So, all of those technocratically-minded ed reformers who thought we should focus on urban charters because test scores showed they “worked” were guilty of mis-judging long-term success based on unreliable short-term measures. The evidence shows that changing test scores is not a particularly good indicator of schools that will improve their students’ lives.
City Fund, NACSA, and others who support portfolio management, harbor-mastering, quarterbacking, or whatever marketing term they are using nowadays are once again left trying to explain exactly how they intend to distinguish the “good” charter schools from the “bad” ones better than parents can. But take comfort, their political ineptitude matches their technocratic inclinations so that I expect they will burn through their $200 million without successfully installing and maintaining any multi-sector portfolio management systems. So children will be safe from their falsely-guided superior judgement.