If you mostly care about test scores, private school choice is not for you. Despite the vast majority of randomized control trials (RCTs) of private school choice showing significant, positive test score effects for at least some subgroups of students, some of those gains have been modest and other effects have been null for at least some subgroups. And now we have two RCTs, in Louisiana and DC, showing significant test score declines for at least some subgroups and in some subjects. The Louisiana decline is large and across-the-board, but the significant, negative effect in the new DC study appears to be “driven entirely by students in elementary grades not previously in a needs-improvement school.”
People will quibble over why these new DC results showed at least a partial decline. They will note that the prior RCT of DC vouchers showed significant test score gains after three years (although the p value rose to .06 in year four even as the positive estimate remained). They will note that vouchers in DC are worth almost 1/3 as much as the per pupil funding received by DC’s traditional public schools and almost half as much as DC’s charter schools. Imagine how they might do if they received comparable resources (and yes resources can matter if there are proper incentives to use resources productively). They will note that almost half of the control group attended charter schools, so to a large degree this study is a comparison of how students do in vouchers relative to charters.
But these largely miss the point — the benefits of private school choice are clearly evident in long term outcomes, not near-term test scores. In the same DC program that just produced disappointing test score effects, using a voucher raised high school graduation rates by 21 percentage points. Similarly, private school choice programs in Milwaukee and New York City were less impressive in their test score effects than in later educational attainment, where private school students in both cities were significantly more likely to enroll in college.
But if what you really care about is raising test scores, you’d be pushing no-excuses charter schools. Rigorous evaluations, like the one in Boston, show huge test score gains for students randomly assigned to no-excuses charter schools. You don’t even have to have school choice to produce these gains. The same team of researchers showed that schools converted into no-excuses charters as part of a turnaround effort produced similarly big gains for students who were already there and did nothing to choose it. The lesson that a fair number of foundations and policymakers draw is that we don’t need this messy and controversial choice stuff. They believe that they have discovered the correct school model — it’s a no excuses charter — and all we need to do is get as many disadvantaged kids into these kinds of schools as we can, with or without them choosing it.
Unfortunately, no excuses charters don’t seem to produce long-term benefits that are commensurate with their huge test score gains. The Boston no excuses charter study, for example, shows no increase in high school graduation rates and no increase in post-secondary enrollment despite large increases in test scores. It’s true that students from those schools who did enroll in post-secondary schooling were more likely to go to a 4 than 2 year college, but it is unclear if this is a desirable outcome given that it may be a mismatch for their needs and this more nuanced effect is not commensurate with the giant test score gains.
This same disconnect between test scores and later life outcomes exists in several rigorously conducted studies of charter schools, including those of the Harlem Promise Academy, KIPP, High Tech High, SEED boarding charter schools, and no excuses charters in Texas. While of course we would generally like to see both test score gains and improved later life outcomes, the thing we really care about is the later life outcomes. And the near-term test scores appear not to be very good proxies for later life outcomes.
So, what should we think about these new test results from DC vouchers, showing some declines for students after one year in the program? We already know from rigorous research that the program improves later life outcomes, so I don’t think we should be particularly troubled by these test results. It may be that control group students are in schools that will fare as well or better on test score measures. But we should remember that 42% of that control group are in the types of charter schools that other research has shown can produce giant test score gains without yielding much in later life outcomes. And we know that treatment group students are in a program that has previously demonstrated large advantages in later life outcomes.
I understand that many reporters, foundations, and policymakers act like they mostly care about test scores and these new results from DC have them all aflutter. But if people could only step back for a second and consider what we are really trying to accomplish in education, the evidence is clearly supportive of private school choice in DC and elsewhere.
(edited to correct error noted in comments)
Jay P. Greene's Blog 
A correction: negative results were not confined to math. Elementary school students had statistically significant declines in reading scores (it’s Figure 5 on page 14).
If all we care about is long-term outcomes, certainly one near the top of my list to care about would be earnings. High school graduation is interesting too, but it’s a stepping stone and not an end in itself. Raj Chetty and his colleagues have shown that higher test scores correlate with higher earnings. So maybe the public should not care *only* about test scores, but there are sound reasons for them to care about test scores.
The largest grade level in the DC study is kindergarten. Asking a parent of a kindergartener to trust that while their child’s reading and math scores will be lower if they use their voucher it will all work out in twenty years is asking quite a lot.
Thanks for the correction, Mark.
As far as earnings go, we have a lot of strong evidence connecting educational attainment to earnings. We have, as far as I know, only one rigorous study connecting test scores to earnings. And that one study has raised some methodological objections and was limited to low-stakes test scores.
Like I said, we’d like to see both test score gains and stronger later life outcomes. But the positive effects of private school choice on attainment are far more important than the mostly positive effects of those programs on test scores (even with the occasional null or negative result).
“If You Mostly Care about Test Scores, You’re a Moron”
FTFY
I would use “troll person” rather than moron, but that is a post for another day.
No, trolls act like morons on purpose.
(Ronald Reagan voice): “There you go again….”
This past week some of our staff met with folks from the Harlem Childrens’ Zone (HCZ) to get some insight on what they do.
a) Damn it’s hard and not easily replicated, and the post-HCZ outcomes are their next big challenge.
b) Their test scores in middle school math are impressive.
Even their folks said to me, “Well, if students don’t do well in your class yet succeed in HS and beyond, that is a metric of success you should claim.”
Our school has never had KIPPean test scores, but the empirical evidence says their later life outcomes are successful. One takeaway from HCZ is indeed we should be more rigorous data gatherers/analyzers of the later life outcomes of our students.
As I understand it, school choice is to help parents/kids escape failing schools to a better school (private or public). Why don’t we learn what math textbooks/programs and assigned readings were in the school in which students got higher test score? All I’m seeing reported are test scores. Is that what school choice is about–a higher test score?
I think Jay addresses this quite well in his post.
I haven’t seen any details on the content of the school curriculum in studies of “choice,” public or private. Can you quote something you see as relevant?
You asked “Why don’t we learn what math textbooks/programs and assigned readings were in the school in which students got higher test score?” The post answers that question: because choice produces the educational outcomes we want without central government control of curriculum.
I’ve had parents in affluent neighborhoods share honestly that the teachers in their ‘affluent’ school aren’t superior teachers. They admit how much they must compensate for what the teachers aren’t doing, or it’s a norm that you talk to your child in sentences, read to them, take them on trips/into parks/travel. All of these help generate superior test scores. Like any other shopping, a parent needs to go kick the tires to know if a school is a best fit for their child; reading the state test scores of a school is not a complete indicator of the education occurring at a school.
[…] Indeed, the nation has seen a backlash against education reduced to such narrow measures, which may not predict future success. And it may be that people want things out of schooling that simply cannot be easily tested, […]
I haven’t seen any details on the content of the school curriculum in studies of “choice,” public or private. Can you quote something you see as relevant?
To Think Tank West. Maybe the public (especially parents) need information on something ed researchers don’t understand–like the school curriculum–and don’t know how to quantify?
Greg et al, this is not an answer to my question about why math textbooks and assigned readings are not detailed in studies of voucher (or other) programs. “…because choice produces the educational outcomes we want without central government control of curriculum.” You may want only higher test scores. Most parents (at least those I know and knew) wanted to know what is/was in their children’s curriculum.
Maybe the lack of interest in the school curriculum is why “gaps” keep widening?
Jay and I are arguing against using test scores as the main measurement of outcomes; not sure how that could be made more obvious for you.
Parents can and do research what the schools assign, rather than relying on people like Jay and I to do it for them, which is one reason choice works so well while attempts to improve curricula through centralized control generally fail.
My concerns, like those of many parents years ago and now, are about the content of the curriculum. If some of you oppose sole use of test scores, why not tell us about that content? Especially when private choice does NOT seem to work very well (in LA and DC). Or recommend ways to find out and report on it?
I haven’t seen any advocacy or discussion of “centralized” control in any of this. Is that what you think parents want? I’ve known lots of parents over the years, and none of them were interested in it on the dozens of Parents Nights I attended. What they wanted to know was what was in the curriculum. In Brookline, we finally got the School Committee to require all teachers K-12 to set forth their curriculum (and sources of readings) by September–for all to see. All binders were to be in a small, separate room next to the Assoc. Super’s office. We knew how important the curriculum was. Maybe researchers don’t.
As I already said, the reason we don’t study curricula is because parents already know how important it is, and do a better job than we can of evaluating their schools (because they know their kids’ needs and we don’t).
As I’ve written before, I’ve become more convinced about benefits of school choice separate from test score gains. So rowing towards you in a sense.
However, you’ve oft written about how test score gains justify X policy. E.g., vouchers in Florida and effect on failing schools. At least that was how I read it.
Would you say you’re moving in how you see the role of test scores in choice, or did I misread things all along?
You are right in sensing that I have learned more about the limitations of using test scores to determine the quality of schools and programs. That being said, I don’t think test scores are irrelevant. And to be clear, I suspect that no excuses charter schools are producing some good results — I just don’t think they are nearly as impressive as the test scores alone would suggest.
So, I suspect that our thinking is not that far apart on these things. My greater appreciation of the limitations of test scores and the importance of other outcomes just makes me less manic than others who think strong test scores mean programs are great while weak test scores mean programs are horrible.
Jay and Mike G: What are your hypotheses about what is going on?
How are the private schools will lower test score progress but better life outcomes different than higher test score growth no excuses charters?
Both have a choice angle (parents are choosing). That suggests choice is necessary but not sufficient.
Are the curric in private schools broader thus giving kids get a broader (and deeper) understanding of the world beyond reading and math?
Do the privates provide more intanglbles, e.g., channeling Judith Harris, a nucleus peer group with norms important for longer term outcomes? I wonder – is adult driven behavioral norming (what you see in no excuses charters) less durable than peer-group driven norming. That seems plausible to me…
Eager to know what you think.
I don’t fully know, but it could be that charters are more narrowly focused on tested subjects and items because tests are the main factor influencing their permission to operate from authorizers. Private schools do not have the same incentive because their continued operation depends on pleasing parents across a broad set of outcomes, not pleasing state appointed officials with easily monitored measures like test scores.
If charters are more likely to focus narrowly on test scores that could cause those scores to be inflated and lead them to shortchange other subjects and goals that influence long term outcomes like attainment.