When Evidence and Science are Really Just Assumptions and Ideology

December 5, 2016

Doug Harris has a new post that attempts to reply to the many critics of his New York Times op-ed, including me.  In the NYT piece Harris claimed that “one well-regarded study found that Detroit’s charter schools performed at about the same dismal level as its traditional public schools.” He also claimed that the relatively light regulatory approach to Detroit’s charter schools has led to “the biggest school reform disaster in the country.”  He prefers instead the heavy-regulation approach adopted in New Orleans, which he says has produced “impressive” results.  He then suggests that failing to believe these claims is “a triumph of ideology over evidence.”

Several critics, including me, noted that the “well-regarded study” Harris cites actually finds that Detroit charter schools are producing significantly greater gains than traditional public school alternatives — gains that are only slightly smaller than those in New Orleans and greater than in another high-regulation darling, Denver.  But Harris wants to continue posturing as the person backed by science and evidence, while describing his opponents as ideologues.  In his reply to critics in Education Next he uses the word “evidence” 16 times.

So, his defense of the claim that Detroit charters perform “at about the same dismal level as its traditional public schools” must be based on evidence, right? Actually, no.  He provides four arguments to rescue his assertion that “the failure of Detroit charter schools to improve student outcomes” is true despite the fact that the CREDO evidence he cited to make that claim shows otherwise.  Each one of those four arguments is based on assumptions, not evidence.  If you believe all of his assumptions, you might believe his claim that Detroit charter schools really are a disaster. But drawing conclusions that depend completely on assumptions and are contrary to the evidence he cites is certainly not science.  It seems more like a faith-based or ideological exercise.

Let’s consider his four arguments.  First he says, “given the lack of oversight in Detroit and evidence from other cities that some charter schools cherry-pick their preferred students, these results may make Detroit’s charter schools look better than they are.”  Got that?  He has no evidence that Detroit charters are cherry-picking students at all, let alone that they are doing so at a higher rate than in New Orleans, but he nevertheless posits that “if it’s happening, then the charter effects on achievement [in Detroit] are inflated.”

Second, given Harris’ assumed concerns about cherry-picking in Detroit charter schools and the inability of the CREDO study to account for that, he examines evidence from the urban NAEP test and finds that the city of Detroit has experienced below average growth in those scores in recent years.  Using NAEP results from the entire city to draw conclusions about Detroit’s charter schools requires a host of assumptions.  He’d have to assume that test results from all schools, charter and traditional, somehow speak to the effectiveness of charter schools.  He’d have to assume that demographic and other non-school factors in Detroit do not affect the comparison of test growth in Detroit relative to other cities.

A number of education analysts have coined a term — misNAEPery —  to capture how unreasonable it is to make the assumptions required to use NAEP to compare policies across jurisdictions.  Oddly, some of those analysts who like to accuse others of misNAEPery, like Morgan Polikoff and Matt Barnum, have somehow failed to denounce Harris’ use of misNAEPery in both the NYT op-ed and in the Ed Next reply to critics.  Both have even “retweeted” Harris’ new post, so we can assume they’ve read it.

And just to anticipate concerns about my own consistency on the use of NAEP, I think comparisons using NAEP that control for observed demographics are about as convincing as the CREDO results, which also rely on comparisons controlling for some observed characteristics.  Information from NAEP or CREDO can be interesting or suggestive, which is why I say that Arizona charters “appear” to be doing very well, even if I am not convinced that any of this is causal.  At the very least, it is useful to offer a disclaimer that neither NAEP nor CREDO provides convincing causal evidence, even if confession does not assure absolution.  Harris does not offer any disclaimer and instead uses his NAEP comparisons to bolster his assumption that Detroit charters may be cherry-picking.

Third, Harris builds on the observation that Detroit city has very low NAEP test scores to assume that “the extraordinarily low standing of the city as a whole, to the degree it is caused by low performance of traditional public schools, should make it easier to improve student outcomes when trying something new.”  It is also quite plausible — perhaps more plausible — that a city with extraordinarily low test scores also has severe social and economic problems that are outside of the control of schools, which would make it harder for charters to improve outcomes.

Lastly, Harris argues that even if Detroit charters have produced gains, the gains are smaller than those in New Orleans, so we should prefer the regulatory approach used in New Orleans to the one in Detroit. But this assumes that any greater gains produced by New Orleans charter schools are caused by the regulatory approach in that city.  In fact, we have no idea whether New Orleans’ regulations helped, hurt, or had no effect on how large the gains in that city were. For all we know, the gains made by New Orleans charters are largely attributable to the importation of top-notch human capital from elite colleges and a huge increase in per pupil spending, and that these gains were made despite the hindrance of burdensome regulations.  The heavy regulations in Denver somehow failed to produce gains as large as those in Detroit.

Other than these four-assumption-dependent arguments, Harris offers very little to defend the strong claims he made in the NYT that Detroit charter schools have been a “failure” and a “disaster.”  He does cite some national “charter-friendly” organizations, like CRPE and NACSA,  as being critical of Detroit charters.  But that is an argument from authority — not evidence.

He also falsely suggests that Detroit has failed to close failing charter schools.  In fact, 30% of Detroit’s charter schools have been shuttered.  And another national charter-friendly organization, NAPCS, gives Michigan higher marks on closure than Louisiana, noting that Michigan has closed more charter schools than Louisiana, 47 to 26.

Harris also cites two voucher studies with negative results — one of which is an RCT and the other not — as proof that lower regulation approaches are less effective.  Leaving aside the distinct possibility that the negative RCT result for Louisiana was actually a function of the over-regulation of that program, it’s important to note that Harris fails to cite the entire literature of rigorous studies on the effects of private school choice programs, which overwhelmingly shows positive outcomes.

I agree with Doug that ideology, which could more kindly be described as “principles” or “values,” has an important role to play in policymaking.  I just disagree with him that the evidence clearly shows the superiority of a high-regulation approach to school choice.  I think it’s more accurate to say that the evidence is unclear on this matter, which means that we may — appropriately — need to rely more on our values, principles, and broader ideology when deciding how to proceed.

[Edited to remove tangential comment that was poorly phrased.]


Putting Kids Before Politics

December 2, 2016

(Guest Post by Jason Bedrick)

Wise words from Rev. Alfred Cockfield II of Brooklyn:

Many members of our congregation live in neighborhoods where their local schools are failing. These parents care deeply about having more than one public school option because it means the difference between their child getting a quality education or just getting passed along.

President Obama and Secretary of Education John King have been strong proponents of public charter schools. They understand what parents in our congregation understand: that public charter schools and school choice are vital to closing the racial achievement gap.

And perhaps surprisingly, so does Donald Trump. For all of his rhetoric during the campaign, he is a strong supporter of public charter schools — and believes the status quo must change.

In the months ahead, we must remain vigilant, and be prepared to challenge the president-elect whenever he proposes a policy that threatens our communities. But we can’t let that stop the progress we know we can make and must make. And nowhere is the need to push forward together clearer than with charter schools.

If President-elect Trump is truly willing to support policies that give the highest-need children the school options that they desperately need, we should be willing to listen. That does not mean compromising our values — rather, it means authentically representing the needs of our communities, whether fighting back against a policy we disagree with or advocating for one we believe will help our children.

That’s the balancing act we face moving forward. And we cannot pretend it will be easy. But we owe it to our children to attempt to navigate the next four years with eyes wide open, and with their interests always front and center.

 


The Ministry of Truth

December 1, 2016

338bb4598fd1e27d058b30ef71ea2f42

(Guest Post by Jason Bedrick)

I’m just going to leave this here:

screen-shot-2016-12-01-at-8-26-39-am


DeVos and the Education Wars

November 29, 2016

(Guest Post by Jason Bedrick)

President-Elect Trump’s nomination of school choice champion Betsy DeVos has become the latest battleground in both the war between pro- and anti-school choice forces as well as the internecine battle between technocratic reformers and market-oriented reformers within the school choice camp. Jay’s take today is a must-read piece. I also added my two cents over at Cato-at-Liberty, defending market-oriented school choice policies from what I see as unfair attacks from the technocrat crowd while simultaneously cautioning my compatriots against pushing for a federal school choice program (e.g., Title I portability). Here’s a taste:

At the center of the panic over Trump’s nomination of DeVos is their support for school choice. Although light on details, Trump has pledged to devote $20 billion to a federal voucher program. As is so often the case, the most vocal opponents of federal school choice are right for the wrong reasons. Not only does the federal government lack constitutional jurisdiction (outside of Washington, D.C., military installations, and tribal lands), but a federal voucher program poses a danger to school choice efforts nationwide because a less-friendly future administration could attach regulations that undermine choice policies. Such regulations are always a threat to the effectiveness of school choice policies, but when a particular state adopts harmful regulations, the negative effects are localized. Louisiana’s folly does not affect Florida. Not so with a national voucher program. Moreover, harmful regulations are easier to fight at the state level than at the federal level, where the exercise of “pen and phone” executive authority is increasingly (and unfortunately) the norm.

The technocratic crowd wants to blame the mediocre results in the charter sector in Michigan (DeVos’s home state) on its supposedly “unregulated” and “laissez-faire” environment, which raises the question: Do they do know what those terms mean? As I note:

Charter schools in Michigan and Arizona may be subject to fewer government regulations than in other states, but it’s absurd to describe the sectors as “laissez-faire” or “an unregulated free market.” For example, charter school regulations in both states, as elsewhere, limit the ability of charter schools to set their own mission (e.g., they must be secular), mandate that they administer the state standardized test, forbid them from setting their own admissions standards, forbid them from charging tuition, limit who can teach in the schools, limit the growth of the number of schools, and so on.

“Laissez-faire” indeed!

Moreover, as JayBlogger Matt Ladner has frequently pointed out, in the real “Wild West” of Arizona, charter schools are knocking the socks off their district counterparts and showing greater improvement than any state average on the NAEP.

Anyway, while we’re on the topic of Trump and education reform, I’d like to express full-throated agreement with Greg Forster’s two recent posts on bigotry and the choices before us, particularly this:

Trump will be president. All of us who work on policy issues have to live in a world where Trump is president. It’s not necessarily a good idea for every decent person to shun him; that means government will be run by scoundrels like Trump.

Every movement needs its Vaclav Klauses as well as its Vaclav Havels – people who are willing to hold their noses and work for a corrupt regime. You simply can’t get anything done otherwise, because there are no non-corrupt regimes.

Milton went to Chile and advised Pinochet. When challenged, he said: “I gave him good advice.”

But if they forget to hold their noses, if they think the regime is good, the movement dies. And they will forget if no one plays Vaclav Havel and goes to jail for telling the truth about the regime.

My biggest fear is that the school choice issue will become tied to Trump. It can never be said too many times: Donald Trump is a notorious racist who discriminates against blacks in his businesses, said a judge of Mexican ancestry couldn’t judge him impartially, constantly flirted with the alt-right, and refused, three times, to repudiate the KKK when first asked to do so. (Just in case this is unclear, the KKK is a criminal organization that murders people and exists to make war on the US government in the name of white nationalism. If Trump wants to learn more about it, he can ask his attorney general, who had a Klan leader executed.)

We in the school choice movement have spent a generation building bridges between the conservatives and libertarians traditionally associated with the issue and progressives and ethnic minority communities. We can’t afford to throw all that away.

Israeli Prime Minister Yitzchak Rabin once said that he would “fight terrorism as if there is no peace process and pursue peace as if there is no terrorism.” We need a similar approach. We should pursue education reform regardless of the Trump administration’s positions on other issues — as Derrell Bradford’s moving personal account reminds us, the stakes are just too high not to. That will entail, at times, working with anyone at the Trump administration who is willing to listen, and supporting good and decent people who go to work for the administration. However, it also means calling out Trump and/or his administration when they do wrong (like, say, Tweet that people should go to jail for engaging in constitutionally protected speech, to take just one example from the last 24 hours), no matter what progress they have made on education reform.

Navigating the political waters over the next four years will be difficult. Even Odysseus only had to pass between Scylla and Charybdis once. I suspect education reformers will find themselves in the straits on numerous occasions in these coming days. I pray that we will have the wisdom to know and the fortitude to do the right thing.


You Can’t Regulate Quality If You Can’t Predict Quality

November 29, 2016

School choice is on the march.  Even before Trump’s election and his selection of Betsy DeVos as Secretary of Education, charter and private school choice have been steadily expanding every year.  Expanding school choice, like almost all of education reform, occurs in the states, so who is in charge in DC will not make too much of a difference other than turning a headwind into a tailwind.

But there is a certain crowd whose lives and thoughts revolve around what is happening in DC, so they have become all aflutter with either hope or dread about the prospects of a Trump presidency for school choice.  Those motivated by dread have fired first in the looming Regulation War, arguing that choice only works — if it works at all — within a heavy regulatory framework to ensure quality.  If choice can’t be stopped at least it can be controlled and that control can improve the choices parents will make.

The problem with the pro-regulation argument is that you can’t ensure quality if you don’t have the ability to predict quality.  It’s true that parents have a hard time anticipating how different schools might affect long-term outcomes and are quite likely to make mistakes in choosing schools.  And some emphasize this fact to justify a strong role for regulators in controlling the range and type of options from which parents can choose.  But what they rarely consider is whether regulators are any better at anticipating how schools will affect long-term outcomes.  After all, regulators can’t protect families against making mistakes if they are equally or more likely to make those mistakes themselves.

I’ve written previously about the disconnect between near-term test score gains and changes in later life outcomes.  If we can’t reliably use rigorously identified test score gains to predict later life outcomes, then on what basis will regulators be able to judge quality to protect families against making bad choices?  And the situation is even worse because most regulators making decisions about what choice schools should be opened, expanded, or closed are not relying on rigorously identified gains in test scores — they just look primarily at the levels of test scores and call those with low scores bad.  Of course, all this does is punish and discourage schools from attempting to serve the most challenging students since the level of scores is more a reflection of student background than school quality.

But let’s say you don’t believe me about the weak predictive power of test score gains and are determined to use tests as the main indicator of school quality.  We are still left with the question of whether regulators are any good at identifying which schools will contribute to test score gains.  Fortunately, we have a recent study that examined whether the criteria used by regulators in New Orleans are predictive of test score growth — even if we accept test gains as a reliable indicator of quality.  The bottom line is that none of the factors used by authorizers to open or renew charter schools in New Orleans were predictive of how much test score growth these schools could produce later on.

In particular, the study examines ratings derived from criteria favored by the National Association of Charter School Authorizers (NACSA) to see if they are predictive of test score growth or enrollment growth.  Neither the NACSA aggregate rating nor individual factors, like school governance features, educational plans, or financial and enrollment characteristics were predictive.  As the study concluded, “None of the factors, however, predict the main dependent variables of interest: SPS, value-added, enrollment, and enrollment growth, as direct measures of future school performance. ”

If regulators are unable to predict which schools will be good (assuming, falsely, that test score gains are a reliable indicator of good schools), how are they supposed to “protect” parents from making bad choices about schools?  The case for regulation is often expressed as a tautology: people will make better choices if they only have good schools from which to choose. But this presupposes that regulators can identify and eliminate bad choices.  And if regulators really could do this, why have school choice at all?  Why not just ensure that every student succeeds by forbidding any school from being bad?

No one should be surprised that NACSA’s criteria have no relationship to their own metric for school quality — test score growth — given how well Arizona charter schools appear to be doing even while NACSA gives the state a very low score for charter quality.  What might be more surprising is that the recent study I’ve been referencing that finds no connection between the criteria regulators use and future test score growth was co-authored by none other than Doug Harris.  Yes, that’s the same Doug Harris who wrote the NYT op-ed claiming that the lack of regulation in Detroit contributed to “the biggest school reform disaster in the country,” while New Orleans’ more heavily regulated approach produced “impressive” results.

As Ramesh Ponnuru has noted, using the same CREDO research that Doug cites, Detroit charter schools produce significantly better outcomes than the traditional public school alternatives.  In fact, the test score gains achieved by Detroit charters exceed those in heavily regulated Denver and only slightly lag those in New Orleans.  (See the table above for a graphic presentation of results across cities).

Mind you, I do not believe the CREDO results because that research only matches on observed characteristics and therefore does not have a rigorous identification of causal effects.  But Doug seems to believe that research even though it undermines the very argument he was making.  And one can only assume that Doug believes his own research showing that regulators lack effective tools to identify and predict good schools.  Doug accuses DeVos of being guilty of “a triumph of ideology over evidence,” but it seems like there is a lot of that going around.

 


Do tax-credit scholarship programs drain money from district schools?

November 15, 2016

(Guest Post by Marty Lueken)

Perhaps the most-cited criticism of educational choice policies is that they “siphon” resources from district schools. High-quality research reveals numerous benefits stemming from educational choice, including benefits for participants, positive competitive effects, and fostering civic values. Policymakers, however, are concerned about the fiscal impact on district schools. A new report, The Tax-Credit Scholarship Audit, finds that such claims about “siphoning” are unfounded.

This report follows up on an earlier report, The School Voucher Audit, which examined the fiscal impact of school voucher programs. In the report, I estimated the fiscal impact of 10 tax-credit scholarship programs in seven states (AZ, FL, GA, IA, IN, PA, and RI) on state governments, state and local taxpayers, and school districts combined. There are currently 21 tax-credit scholarship programs operating in 17 states. Programs with less than three years of data were not included in the analysis. This sample includes the largest tax-credit scholarship programs in the country. These programs accounted for 93 percent of all scholarships awarded in 2013-14, the final year included in the analysis.

Here’s a summary of the results:

Participation

Since the first tax-credit scholarship program was established in 1997, more than 1.2 million scholarships were awarded to students through 2014. Participation in tax-credit scholarship programs has grown every year since 1997. Participation in these programs grew, on average, by 20 percent each year since 2000.

Screen Shot 2016-11-15 at 3.24.05 PM.png

Tax-credit scholarship programs require time to establish. Looking at the programs by year in operation, the number of scholarships awarded more than tripled after their fifth year in operation, from 28,000 to more than 95,000.

Screen Shot 2016-11-15 at 3.25.29 PM.png

Educational choice programs are popular and continue to expand, though not to the detriment of the current district school system, as critics argue. Even in states where eligibility for school choice programs is expansive, we haven’t seen a mass exodus of students from district schools. Moreover, the best evidence on the effects of school choice on district schools shows that district schools improve in response to competition.

Fiscal Impact

Depending on assumptions about the number of students switching from district or charter schools to private schools and the share of students who receive multiple scholarships, I estimated that these 10 tax-credit scholarship programs saved taxpayers between $1.7 billion and $3.4 billion since Arizona enacted the first such program in 1997. That equates to between $1,650 and $3,000 in savings for each scholarship recipient.

screen-shot-2016-11-15-at-3-26-42-pm

This paints a starkly different picture from the “siphoning” argument that school choice critics like to make. Context is useful when discussing the fiscal impacts of these programs. For all the controversy that sometimes surrounds school choice programs, the fact is that these programs make up a very small share of states’ K-12 revenues. The cost of the taxpayer subsidies for the programs covered in the report range from 0.1 percent each for Rhode Island and Indiana’s programs to 1.5 percent for Arizona’s four programs.

Screen Shot 2016-11-15 at 3.27.32 PM.png

When critics claim these programs are costly or drain funds, they usually focus on just the cost of the taxpayer subsidy without giving any consideration to savings. But that tells only one part of the story. It’s also true that districts are no longer responsible for educating students that leave via these programs. In the short-run, they incur variable cost savings. My analysis accounts for this and even takes a more conservative approach to estimating variable costs than in other work by economists.

These savings don’t usually show up in budget reports, however, though public officials make choices—whether implicitly or explicitly—about what they do with the savings. As I note in the report:

“When school choice programs are enacted, however, it is usually the case that savings do not automatically materialize as reductions in K–12 expenditures because public officials must actively make decisions to reduce such expenditures. When students leave public schools, officials have more room in their budgets to allocate resources for educating students that remain in those schools.” (pp. 15-16)

Government officials have many options regarding what to do with these savings. They can reinvest them in district schools, which means more resources for the fewer number of students who remain in them, or they can also reallocate those savings to other public services. They can also choose to hold all or some of the funds rather than spend them. They can subsequently lower property taxes or save and invest these funds. Typically, however, it’s not clear how these freed-up resources are used.

It is a worthwhile policy goal to provide and expand quality educational options so that parents can match their children to the education provider that works best for them. Tax-credit scholarship programs offer one way to help achieve this goal by providing avenues for states to attract capital and invest in their state’s education system without harming taxpayers or students.

 

Martin F. Lueken, Ph.D. is the Director of Fiscal Policy and Analysis at EdChoice.


Evidence for the Disconnect Between Changing Test Scores and Changing Later Life Outcomes

November 5, 2016

Over the last few years I have developed a deeper skepticism about the reliability of relying on test scores for accountability purposes.  I think tests have very limited potential in guiding distant policymakers, regulators, portfolio managers, foundation officials, and other policy elites in identifying with confidence which schools are good or bad, ought to be opened, expanded, or closed, and which programs are working or failing.  The problem, as I’ve pointed out in several pieces now, is that in using tests for these purposes we are assuming that if we can change test scores, we will change later outcomes in life.  We don’t really care about test scores per se, we care about them because we think they are near-term proxies for later life outcomes that we really do care about — like graduating from high school, going to college, getting a job, earning a good living, staying out of jail, etc…

But what if changing test scores does not regularly correspond with changing life outcomes?  What if schools can do things to change scores without actually changing lives?  What evidence do we actually have to support the assumption that changing test scores is a reliable indicator of changing later life outcomes?

This concern is similar to issues that have arisen in other fields about the reliability of near-term indicators as proxies for later life outcomes.  For example, as one of my colleagues noted to me, there are medicines that are able to lower cholesterol levels but do not reduce — or even may increase — mortality from heart disease.  It’s important that we think carefully about whether we are making the same type of mistake in education.

If increasing test scores is a good indicator of improving later life outcomes, we should see roughly the same direction and magnitude in changes of scores and later outcomes in most rigorously identified studies.  We do not.  I’m not saying we never see a connection between changing test scores and changing later life outcomes (e.g. Chetty, et al); I’m just saying that we do not regularly see that relationship.  For an indicator to be reliable, it should yield accurate predictions nearly all, or at least most, of the time.

To illustrate the un-reliability of test score changes, I’m going to focus on rigorously identified research on school choice programs where we have later life outcomes.  We could find plenty of examples of disconnect from other policy interventions, such as pre-school programs, but I am focusing on school choice because I know this literature best.  The fact that we can find a disconnect between test score changes and later life outcomes in any literature, let alone in several, should undermine our confidence in test scores as a reliable indicator.

I should also emphasize that by looking at rigorous research I am rigging things in favor of test scores.  If we explored the most common use of test scores — examining the level of proficiency — there are no credible researchers who believe that is a reliable indicator of school or program quality.  Even measures of growth in test scores or VAM are not rigorously identified indicators of school or program quality as they do not reveal what the growth would have been in the absence of that school or program.  So, I think almost every credible researcher would agree that the vast majority of ways in which test scores are used by policymakers, regulators, portfolio managers, foundation officials, and other policy elites cannot be reliable indicators of the ability of schools or programs to improve later life outcomes.

With the evidence below I am exploring the largely imaginary scenario in which test scores changes can be attributed to schools or programs with confidence.  Even then, the direction and magnitude of changing test scores does not regularly correspond with changing later life outcomes.  I’ve identified 10 rigorously designed studies of charter and private school choice programs with later life outcomes.  I’ve listed them below with a brief description of their findings and hyperlinks so you can read the results for yourself.

Notice any patterns? Other than the general disconnect between test scores and later life outcomes (in both directions), I notice that the No Excuses charter model that is currently the darling of the ed reform movement and that New York Times columnists have declared as the only type of “Schools that Work” tend not to fare nearly as well in later outcomes as they do on test scores.  Meanwhile the unfashionable private choice schools and Mom and Pop charters seem to do much better on later life outcomes than at changing test scores.  I don’t highlight this pattern as proof that we should shy away from No Excuses charters.  I only mention it to suggest ways in which over-relying on test scores and declaring with confidence that we know what works and what doesn’t can lead to big policy mistakes.

Here are the 10 studies:

  1. Boston charters (Angrist, et al, 2014) – Huge test score gains, no increase in HS grad rate or postsecondary attendance. Shift from 2 to 4 yr
  2. Harlem Promise Academy (Dobbie and Fryer, 2014) – Same as Boston charters
  3. KIPP (Tuttle, et al, 2015) – Large test score gains, no or small effect on HS grad rate, depending on analysis used
  4. High Tech High (Beauregard, 2015) – Widely praised for improving test scores, no increase in college enrollment
  5. SEED Boarding Charter (Unterman, et al, 2016) – same as Boston charters
  6. TX No Excuses charters (Dobbie and Fryer, 2016) – Increase test scores and college enrollment, but no effect on earnings
  7. Florida charters (Booker, et al, 2014) – No test score gains but large increase in HS grad rate, college attendance, and earnings
  8. DC vouchers (Wolf, et al, 2013) – Little or no test score gain but large increase in HS grad rate
  9. Milwaukee vouchers (Cowen, et al, 2013) – same as DC
  10. New York vouchers (Chingos and Peterson, 2013) – modest test score gain, larger college enrollment improvement