Actually, it was Friedrich Hayek who said this, but this image seemed a lot funnier. Hayek — the economist with the funny mustache, not the movie star — was briefly a professor at the University of Arkansas before going on to the University of Chicago and winning a Nobel Prize. Go Hogs! As far as I know, Salma Hayek has never taught at the University of Arkansas and has also never won a Nobel Prize. I think we can all see the causal connection.
While we are on the topic of the difficulty of drawing causal connections and being reminded of how little we know about what we imagine we can design, I’d like to add another piece of evidence to my previous post on “The Over-Confidence of Portfolio Management.” The central idea of portfolio management is to identify and close under-performing schools while opening and expanding higher quality schools.
The problem I noted in my previous post, is that portfolio management (PM) does not have a reliable tool for identifying which are the “good” and “bad” schools. In New Orleans they identify “bad” schools based on the state’s school grading system, which is determined almost entirely by the level of student achievement on math and reading tests (and almost not at all on the growth in learning). There is no reason to believe that schools with low levels of scores are “bad” schools, so there is no reason to believe that portfolio management is effectively identifying and closing bad schools. And if PM in NOLA can’t effectively distinguish between good and bad schools, there is no reason to believe that whatever progress has been made in New Orleans is due to portfolio management.
But this is not holding back PM supporters from adopting portfolio management as the “best practice” and hoping to implement it in Detroit and elsewhere. They’re confident that they can identify bad schools for closure by focusing on growth in standardized test results, even if that was not what was done in NOLA. The problem I noted in my last post is that even annual growth in test performance does not seem to be strongly predictive of later-life success. Looking at 7 rigorous studies of long-term outcomes for charter and private school choice programs, we find that programs that produce large test score gains produce little or no gain in educational attainment while other programs with modest or no gains in test scores make large gains in high school graduation, college attendance, and even workplace earnings.
Well, now we have an 8th rigorous study and it fits the same pattern as the previous 7. Joshua M. Beauregard’s Harvard doctoral thesis examines college outcomes for students applying by lottery for admission to High Tech High (HTH) School in San Diego. High Tech High is a widely-lauded charter school that receives significant financial support from many of the big education reform foundations. It’s a model of the kind of school that portfolio managers are supposed to open and expand.
So what is the effect of HTH on whether its students attend college? Beauregard finds: “I detect no impact on the probability of enrollment in college of any type, which includes two-year and four-year colleges, for my aggregate sample. My estimates are consistent when including baseline demographics and ELA scores.” (p. 28) Similar to Angrist, et al’s results from studying “no excuses” charter schools in Boston, Beauregard does find that attending HTH seems to shift college-going students from two to four-year colleges, but it has no overall effect on post-secondary education. He describes the increase in four-year college attendance as “statistically significant” although the results in Table 5 show that this is only at p<.1. And we have no idea whether shifting students from 2 to 4 year colleges is good if students are unable to complete a 4 year degree. Beauregard also reports no increase in attendance at “competitive” four-year colleges as a result of attending High Tech High.
With HTH we have another example of a school of choice with impressive test score performance that does not produce commensurate improvements in long-term outcomes for students. The central flaw of portfolio management is that the tool it uses to identify which schools should be closed and which should be opened and expanded is not well-connected with long-term success for students. PM can’t improve outcomes for students by closing “bad” schools if it can’t accurately determine which are the bad schools.
Maybe Salma Hayek should explain to them “how little they really know about what they imagine they can design.”
The only thing that could improve this already awesome post is an image of Satanica Pandemonium…
A supplemental lesson here is that the evidence can change. I remember a decade ago when we were saying test scores were a good metric because they correlated with life outcomes. But the evidence we had then was a lot less (and a lot less good) than the evidence we have now.
So “evidence based” conclusions (whatever the heck that even means) should not be idolized and treated as permanently and unquestionably right.
[…] outcomes. I’ve written about this at greater length elsewhere (see here and here), but we have eight rigorous studies of school choice programs in which the long-term outcomes of […]
[…] I’ve described at least 8 studies that show a disconnect between raising test scores and stronger later life outcomes. […]
[…] life outcomes. [SEE https://jaypgreene.com/2016/05/02/the-weak-predictive-power-of-test-scores/ AND https://jaypgreene.com/2015/11/14/more-on-the-over-confidence-of-portfolio-management/ ] Well, now we have a 9th. Earlier this month MDRC quietly released a long-term randomized […]
[…] and college attendance rates, college completion rates, and even higher employment and earnings. I’ve described at least 8 studies that show a disconnect between raising test scores and stronger later life […]
This will help it.