I’ve written several times recently about how short term gains in test scores are not associated with improved later life outcomes for students. Schools and programs that increase test score quite often do not yield higher high school graduation or college attendance rates. Conversely, schools and programs that fail to produce greater gains in test scores sometimes produce impressive improvements in high school graduation and college attendance rates, college completion rates, and even higher employment and earnings. I’ve described at least 8 studies that show a disconnect between raising test scores and stronger later life outcomes.
Well, now we have a 9th. Earlier this month MDRC quietly released a long-term randomized experiment of the effects of the SEED boarding charter school in Washington, DC. Because SEED is a boarding school, there was a lot of hope among reformers that it might be able to make a more profound difference for very disadvantaged students by having significantly more time to influence students and structure their lives. Of course, boarding schools also cost significantly more — in this case roughly twice as much as traditional non-residential schools.
While the initial test score results are very encouraging, the later life outcomes are disappointing. After two years students admitted to SEED by lottery outperformed those denied admission by lottery by 33% of a standard deviation in math and 23% in reading. If we judged the quality of schools entirely based on short term changes in test scores, as many reformers would like to do, we’d say this school was doing a great job.
In fact, SEED may be doing a great job in a variety of ways, but when we look at longer term outcomes for students on a variety of measures the evidence demonstrating SEED’s success disappears or even turns negative. Of the students accepted by lottery to SEED 69.3% graduate from high school after four years compared to 74.1% for the control group, a difference that is not statistically significant. And when asked about their likelihood of attending college, there was no significant difference between the two groups. SEED students also score significantly higher on a measure of engaging in risky behavior and lower on the grit scale.
We’ve seen this pattern before. Research by Marty West and colleagues of no excuses charter schools in Boston found large gains in test scores but also significantly lowered student performance on noncognitive measures. And Josh Angrist and colleagues found that those schools actually decrease four year high school graduation rates despite large gains in test scores. In their words:
Perhaps surprisingly given the gains in test score graduation requirements reported in column 2 of table 4, the estimates in column 4 of this table suggest not. In fact, charter attendance reduces the likelihood a student graduates on time by 14.5 percentage points, a statistically significant effect.
It’s time that people start paying a lot more attention to this pattern of a disconnect between short term test score gains and long term life outcomes. We can’t just dismiss this pattern as fluke. And the reduction in noncognitive skills may be important for explaining this pattern. Reduced grit scores may not just be the product of reference group bias. It appears that certain types of charter schools that are able to produce large test score gains also lower character skills and fail to yield long term improvements in life outcomes. Conversely other types of charter and private schools in choice programs fail to improve test scores but yield large gains in later life outcomes.
If we think we can know which schools of choice are good and ought to be expanded and which are bad and ought to be closed based primarily on annual test score gains, we are sadly mistaken. Various portfolio management and “accountability” regimes depend almost entirely on this false belief that test scores reveal which are the good and bad schools. The evidence is growing quite strong that these strategies cannot properly distinguish good from bad schools and may be inflicting great harm on students. Given the disconnect between test scores and later life outcomes we need significantly greater humility about knowing which schools are succeeding.
Given the odd way in which we create these tests — they tend to attempt to measure “skills” and not knowledge — these findings are not surprising. As E.D. Hirsch has written, “General knowledge is… the best single predictor of later academic achievement among preschoolers and kindergartners, as has been shown by analyses of the Early Childhood Longitudinal Survey–Kindergarten Cohort (ECLS-K1992)” (http://educationnext.org/primer-on-success/).
Is any state considering a policy where they explicitly link charter high school “renewal” to 3 outcomes?
Ie, admit age 14, measure age 24 on –
Consumption of Gov’t Services
Net Worth (particularly to notice college debt)
I think a number of charters would pivot if that were in place!
-Less “bad fit” college for all
-Paying lots of attention to helping kids avoid bad college debt – b/c that would kill them on net worth metric
-More family planning, character ed
-More vocational – particularly placement (vs training)
-More openness to military service
Hi Mike — I don’t think any states are considering these as explicit criteria. Given the delay before these outcomes are observed, we may have to rely more on subjective judgment and less on a mechanistic/legalistic checklist of outcomes.
[…] Jay Greene: […]
The methods employed by charters to get higher test scores often focus on rigid compliance in a militaristic social environment, combined with a curriculum narrowly focused on test preparation. Future success is better predicted by students who have exercised an element of choice and autonomy in their learning and a broad curriculum focused on literacy, numeracy, social studies, science, and the arts. In other words, a quality public education.
[…] is what Professor Greene posted on his blog on Tuesday: “I’ve written several times recently about how short term gains in test scores are […]
[…] A recent evaluation of the the SEED School in DC again shows schools that raise test scores don’t always improve students’ later-in-life outcomes. […]
[…] The Disconnect Between Changing Test Scores and Changing Later Life Outcomes Strikes Again […]
[…] of people, including both school choice advocates and education reform opponents, say there’s little evidence that standardized test score gains in math and reading lead to improved long-term life outcomes. […]
[…] especially so given that the No Excuses charter model that has become the darling of ed reformers often comes up short at improving later life outcomes, while private school choice programs seem to fare better at improving high school graduation, […]
[…] be opened, expanded, or closed, and which programs are working or failing. The problem, as I’ve pointed out in several pieces now, is that in using tests for these purposes we are assuming that if we can […]