NAEP Gains by Spending Trend- Some States are the Harlem Globetrotters, Others the Washington Generals

February 24, 2016

(Guest Post by Matthew Ladner)

Notice the large number of states on the negative side of zero on the spending bar. In the immortal words of Lee Ving it’s already started.


Once again let me note with insufferable state pride that Arizona is your ROI champion doing things that no one would have thought possible.

Go majority-minority, cut funding and improve scores? No problem-nothing but net!

So it is tough to pick a Washington Generals, but I’ve got mine narrowed down between Alaska, New York and Wyoming.


Harlem Kids Go To College: Another Positive Charter School Study

August 5, 2013

(Guest Post by Collin Hitt)

Harlem Promise Academy is a charter middle school, part of the Harlem Children’s Zone. Previous studies from Will Dobbie and Roland Fryer have found big test score gains. A new paper by the Harvard research pair finds that the school had large impacts on college attendance, even larger than the previous gains in test scores would have indicated. From their new paper:

Attending the Promise Academy increases the probability of enrolling in college by 24.2 (9.7) percentage points, an 84 percent increase. In Appendix Table 2, we show that lottery winners are also 21.3 (5.9) percentage points more likely to attend a four-year college and 7.2 (2.3) percentage points less likely to attend a two-year college.

The charter school not only increases the likelihood that its students will attend college, but it increases the quality of the colleges that they attend. Harlem Promise Academy is considered an exceptional school in many minds because of its inclusion in the larger HCZ neighborhood experiment, which includes “wrap-around” social services meant to address issues of poverty. So Dobbie and Fryer collected lottery records at three other charter schools across the country that don’t feature HCZ-style community services, including Noble Network in Chicago, a personal favorite of mine. They found similar college enrollment gains.

They also tested whether the Promise Academy had an impact on lifestyle choices. Charter enrollment appeared to lower teen pregnancy rates by 71 percent and, for boys, drove the observed incarceration rate to almost zero.

They close with what I think is a crucial point for the academic community and the education reform movement to understand:

The education reform movement is based, in part, on two important assumptions: (1) high quality schools can increase test scores, and (2) the well-known relationship between test scores and adult outcomes is causal. We have good evidence that the rst assumption holds (Angrist et al. 2010, Abdulkadiroglu et al. 2011, Dobbie and Fryer 2011a). This paper presents the first pieces of evidence that the second assumption may not only be true, but that the cross-sectional correlation between test scores and adult outcomes may understate the true impact of a high quality school, suggesting that high quality schools change more than cognitive ability. Importantly, the return on investment for high-performing charter schools could be much larger than that implied by the short-run test score increases.

As discussed on this blog, there is now a litany of gold-standard studies of charter schools that find test score gains. Perhaps these studies provide only a glimpse of the benefits to come. We don’t know yet, which is why Dobbie and Fryer do what every smart researcher does – they call for more research.

For now, we can say one thing: ANOTHER random-assignment, gold-standard study finds impressive gains for charter schools. What is that now, thirteen? It’s actually getting hard to keep track.

P.S. There’s another intriguing finding. Alums of Harlem Promise Academy were given a survey that included Duckworth’s “Grit Scale,” which asked them to self-report their persistence, focus and work ethic. The charter school alums scored far lower than the comparison group.  This suggests that the self-reported Grit Scale may be a bad measure of actual grit, since it suggested the opposite of the grit outcomes that were observed.

Christopher Lee for The Al

October 30, 2021

(Guest Post by Matthew Ladner)

Descended from Charlemagne. Spoke five languages. Served in the British Secret Service in World War II. Related by marriage to Ian Fleming, said to be the inspiration for James Bond, later played a Bond villain. Required to seek the permission of the King of Sweden to get engaged – and received it. Innumerable film credits. Turned down the role of Grand Moff Tarkin but later played a fallen Jedi/Sith Lord. Re-read The Lord of the Rings annually, settled for Saruman after losing out for the role of Gandalf. Cut a heavy metal album at age 90.

Christopher Lee really was the most interesting man in the world. Anyone who can kill so many villains that the files are still sealed and then portray them on the silver screen for decades is Al-worthy in my book!

Happy 90th Birthday, Dr. Sowell!

June 30, 2020

Screen Shot 2020-06-29 at 11.49.16 PM

(Guest Post by Jason Bedrick)

Dr. Thomas Sowell–economist, public intellectual, author of more than 50 books, and living legend–turns 90 today, just days after releasing his latest book, Charter Schools and Their Enemies. Schools of choice like charter schools are, in Sowell’s view, the best way to wipe out educational disparity.

As he notes in his new book, Sowell’s research into what makes schools successful, particularly for minorities, began decades ago. In 1974, The Public Interest published Sowell’s article, “Black Excellence: The Case of Dunbar High School,” on how a school run by members of the black community for children of the black community attained great success.

Sowell experienced such schools firsthand: he grew up in Harlem but got, in his view, a great education. He went on to graduate magna cum laude from Harvard followed by a graduate degree at Columbia and a doctorate in economics at the University of Chicago, where he studied under the great Nobel laureate economist, Milton Friedman. (Although a Marxist at the time, Sowell’s views evolved and he eventually came to support free markets. He and Friedman became friends and Sowell is currently the Rose & Milton Friedman Senior Fellow on Public Policy at the Hoover Institution.) Although relatively poor and persecuted, Sowell’s research found that the black community was able to provide a high-quality education for their young when given the opportunity.

Sowell returned to this theme in a 1981 debate about education on Firing Line with William F. Buckley. Nearly four decades later, it’s amazing how little the objections to school choice have changed. After Sowell proposed empowering families with school choice via any of several mechanisms (vouchers, open enrollment, tuition tax credits), his interlocutor immediately objected that “uneducated” parents would be unable to choose wisely for their children–an objection recently echoed by elitist politicians and professors alike. Sowell punctured this paternalism with a history lesson (how freed slaves who had been forbidden to learn to read ensured that the next generation was educated) and his own personal history. “I think you’d have very few blacks who finished college, including myself,” he explained, “if they had to have college-educated parents to send them there.”

In a 1986 debate (alongside Friedman and Buckley), Sowell addressed an assertion by Al Shanker of the American Federation of Teachers that what the education system needed was basically more of the same just with more money–essentially the same tune AFT has been singing ever since (even though the racial achievement gap that Shanker claimed would narrow has actually widened despite a massive increase in per-pupil spending). Sowell wasn’t buying it, pointing to the narrower racial achievement gap in the private schools. Debater Bill Honig retorted that a study had found that when private and public schools employed the same methodologies, the results were the same. Sowell immediately countered that this proved his point:

I don’t think there’s any magic about the institutional nature, I think the magic is about competition… If you’re saying to me that the private schools and public schools get the same result when they do the same thing, and you’re saying to me that the private schools close the gap between blacks and whites more, then you’re saying to me that the public schools aren’t doing what they should be doing.

In other words: incentives matter. On net, schools that do the same things with similarly situated students will have similar results no matter the sector. What matters is what schools do, and what they do is a function of who decides. As Sowell wrote in Intellectuals and Society, “The most important decision is who makes the decision.” When parents get to decide where their children attend school, their priorities will be reflected in the school system. When politicians and distant bureaucrats decide, the system will reflect their priorities instead. When parents get to decide, change can happen immediately if they decide to switch schools. When politicians and bureaucrats decide, change happens on their timeline.

Then as now, opponents of parental choice want parents to be patient. “Just give us more time and more money and we’ll get this fixed for you right away!” At the end of this exchange, Sowell channeled the frustration and righteous anger of parents with this “wait and see” approach:

You mention these various horrible things that are going to happen… if we have privatization, and that we should never resort to that until we have exhausted all forms of change in the public schools — how many thousands of years will it take to exhaust all possible forms of change in the public schools?

More than three decades later, most families are still waiting.

There are many more choices available today than in the 1980s, but options still remain elusive for too many families, especially low-income families of color. Untold numbers of children are on waitlists for charter schools or scholarships, options that should and would be available but for political opposition.

Fortunately, there are signs of progress. Even with state legislatures being shut down during the COVID-19 pandemic, Utah passed a new tax-credit scholarship for students with special needs and Florida Gov. Ron DeSantis just signed into law the largest expansion of a school voucher program in the nation’s history.

Let us hope that by Dr. Sowell’s 100th birthday, all families will have access to the learning environments that work best for their children.

The Ravitches of Libel

June 19, 2020

Ravitch in the Schoolhouse Door

Diane Ravitch stands in the schoolhouse door to block children of color from accessing a charter school their parents chose.

(Guest Post by Jason Bedrick)

I’ve generally avoided engaging with Diane Ravitch over the last several years as her writing and followers often seem more like a form of primal-scream therapy than substantive discourse. However, her recent smear of Robert Pondiscio and Eva Moskowitz demands rebuttal.

For those who don’t know Pondiscio, he’s one of the most decent men you could meet. Many moons ago, he left a prestigious and lucrative career in media (TIMEBusinessWeek) to teach fifth grade at a public school in the South Bronx serving a very low-income population, mostly people of color. Since then, he has dedicated his life to improving the lot of the less fortunate by advocating for reforms he believes will improve the quality of education and expand opportunities. He still splits his time between his think tank work (he’s a senior fellow at the Thomas B. Fordham Institute) and teaching high school civics at Democracy Prep, a charter school network based in Harlem.

Recently, Pondiscio wrote a book, How the Other Half Learns: Equality, Excellence, and the Battle Over School Choice, for which he embedded himself in Success Academy, a charter network run by Eva Moskowitz that has not only closed the racial achievement gap, it has reversed it. As Thomas Sowell notes in today’s Wall Street Journal, Success Academy’s “predominantly black and Hispanic students already pass tests in mathematics and English at a higher rate than any school district in the entire state, [including the] predominantly white and Asian school districts where parental income is some multiple of what it is among Success Academy students.”

Success’s successes have rankled defenders of the traditional district school system, which looks pretty terrible by comparison. They’ve leveled a host of critiques of varying merit, and Pondiscio himself was not shy about illuminating Success Academy’s warts in his very thoughtful and nuanced book. (You can listen to interviews Pondiscio gave about the book here and here.)

Thoughtfulness and nuance, however, are not Ravitch’s jam. Piling on a recent controversy in which Moskowitz was accused of racism for merely Tweeting about the “horrific, senseless deaths of innocent black men and women” like “George Floyd, Ahmaud Arbery, Breonna Taylor, and too many others who have died for no other reason than the color of their skin” using the hashtag #BlackLivesMatter rather than issuing a press release, Ravitch decided to crank the unhinged libel up a notch on her blog:

The exchange between Moskowitz and a first-year teacher set off a debate about institutional racism in Success Academy and its harsh no-excuses methods. Those draconian disciplinary methods were defended by Robert Pondiscio of the conservative Thomas B. Fordham Institute, who is white, and by Moskowitz, who is also white. Black children need harsh discipline, they argued. [emphasis added]

Pondiscio, naturally, took umbrage at this blatant mischaracterization of his views:

Screen Shot 2020-06-19 at 11.50.19 AM

In response, Ravitch pointed to a Chalkbeat article in which Pondiscio said the following:

But Moskowitz has vigorously defended her network’s strict approach arguing that exacting behavior expectations that are consistently enforced provide a necessary condition for student learning. And network leaders argue it works: Success’ students, the vast majority of whom are Black or Latino, typically outperform much whiter and more affluent districts on state tests. Parents of color continue flocking to Success, and network leaders are honest about what will be expected of them and their children.

“There is no doubt in my mind that there is a significant appetite among low-income parents for exactly the flavor of education that Eva Moskowitz offers,” said Robert Pondiscio, a senior fellow at the conservative-learning Fordham Institute who spent a year observing a Success elementary school in the South Bronx and wrote a book about it. “It just does violence to reality to pretend that this is some kind of pedagogy that’s being imposed on families of color.”

If you’re puzzled as to how anything in that quote can be construed as arguing that “black children need harsh discipline,” that’s because you don’t have access to the Ravitch’O’Matic Meaning Translator™. Fortunately, I have an older model that still works which I have dusted off. I’ve entered the quoted text above and we’re off the races!

  1. “the flavor of education that Eva Moskowitz offers” = “harsh discipline”
  2. “a significant appetite among low-income parents” = “black children are in need of”

If you’re still confused, that’s because you’re sane.

Whether the disciplinary practices at Success Academy and other “No Excuses”-style charter schools are “harsh discipline” or not is a matter of debate, but Ravitch only operates within an ideological echo chamber so she cannot fathom that there are people (like, for example, Pondiscio and most of the families who choose those schools) who disagree with that characterization. But even if the characterization is fair, it’s not fair to portray it as though it were Pondiscio’s characterization. It is not.

Even more inflammatory and unfair is the second of Ravitch’s rhetorical leaps. “Some parents of color choose X” simply cannot be translated as “all children of color must be subject to X” under any sane understanding of the rules of language. This charge is simply insane.

Daniel Willingham tried to gently and patiently explain this to Ravitch in the comments section, but she replied with circular and obtuse nonsense: “I relied on the words in the Chalkbeat article” and whined that Pondiscio took to Twitter to defend himself against her public calumny instead of emailing her. She continued:

I am unsure what Robert objects to. He has my personal email. Why doesn’t he write and tell me what he finds objectionable? Does he oppose “no-excuses” disciplinary policy? Does he think it is not “harsh”? Did he object to my describing it as harsh? I am totally confused about what he wants me to change.

Is she really that obtuse?

Perhaps. She even doubled down with a subsequent blog post. Senior scholars are generally owed some amount of deference and respect, but Ravitch is sacrificing any claims to such deference by her abhorrent behavior. There are at least three possible explanations for her mischaracterization of Pondiscio’s views:

  1. She lacks the basic faculty of language comprehension. If we take her at her word, she honestly cannot decipher statements in plain English and is left absolutely befuddled when people try to explain it to her using small words. Indeed, as has been documented extensively at this blog, her recent “scholarship” ranges from sloppy to shoddy, she is prone to make outlandish and self-aggrandizing statements, and she has even engaged in conspiracy theorizing that’s downright hillbilly nuts. If so, then people shouldn’t take her seriously.
  2. She knows what’s she’s doing and she’s mendacious. If she does actually possess the mental faculties to comprehend basic English, then she is intentionally twisting Pondiscio’s words in a particularly mendacious way. She knows how damaging a charge of “racism” (merited or not) is, especially in this climate, and has no compunction smearing someone simply because they hold different policy views (or, perhaps, because he criticized her book). If so, then people shouldn’t take her seriously.
  3. She is a racist and she is projecting. Pondiscio spends his time trying to empower black families with the ability to choose a high-quality education for their children. Ravitch spends her time, like George Wallace, standing in the schoolhouse door trying to block black families from accessing the schools they want to choose. To deflect attention from her own racism, she points at someone else who holds different policy views and hopes no one will notice. If so, then people shouldn’t take her seriously — and, indeed, she should be run out of polite society.

Ravitch might reasonably object that these three characterizations of her motives are unfair, but all three are infinitely more fair and supported by evidence than Diane’s Ravitchian reading of Pondiscio’s views.

She owes him an apology.

My Apple is Bigger

August 29, 2017

(Guest post by Patrick J. Wolf)

New York City public charters have been much in the news of late (see here & here) for hitting it out of Yankee Stadium on student achievement.  When Judge-ing (sorry, couldn’t resist) The Big Apple’s charter school sector, add this little fact to the case:  charters out-slug their peers at a lower cost.

That is the conclusion of my latest study of charter school funding inequity, co-authored with Larry D. Maloney.  It is fun to study New York City, in part because of great potential for wordplay but also because the place is so darn big that you can disaggregate results by borough and still have district v. charter comparisons informed by large samples.  So, “start spreading the news…”

There are over 1,000,000 public school children in The Big Apple.  Seven percent of them attended charter schools during Fiscal Year 2014, the focus of our study.  Cash revenue to charter schools averaged $15,983 per-pupil while payments to district-run schools averaged a much more generous $26,560 per-pupil.  “You just wait a New York Minute Mr. Henley,” you might caution, “The New York City Department of Education actually provides in-kind services to students in charter schools that represent a funding resource not accounted for in your cash calculations.”  You would be right.  After factoring in the cash value of such in-kind services, charter schools receive a mere $4,888 less in per-pupil funding than district schools (Figure 3).  New York City charters schools are outperforming the City’s district schools at about 81 cents on the dollar.

I’ll admit that my figure isn’t nearly as MoMA-worthy as Matt’s post-modernist depiction of the Arizona school districts that refuse to accept students through inter-district choice, but it makes a crucial point.  Even accounting for the value of everything contributed in support of charter schools in New York City, district schools still get more money per student.

Critics of our prior charter school funding studies (available here and here) have claimed that we are making Big-Apple-to-Big-Orange comparisons, since district schools provide more extensive educational services to students than charters.  Our accounting for in-kind district services to charters fully addresses that argument.  After factoring in the value of co-located facilities, transportation, meals, special education services, health services, textbooks, software, etc., all of which are provided to charters in New York City so that the scope of their services is equal to that of district schools, the charters still receive less funding.  We even examined school spending patterns, in addition to funding patterns, and the story is the same.

Surely the student populations in district schools are needier than those in charter schools, thereby justifying the funding gap, right?  Actually no.  The population of charter school students in New York City contains a higher percentage of free-and-reduced price lunch kids than the population of district school students (Figure 4).

The percentage of students with disabilities is only slightly higher in district schools versus charter schools, 18.2% compared to 15.9%.  That means that districts enroll 21,342 “extra” students with disabilities compared to charters.  For the special education enrollment gap favoring districts to explain the entire funding gap favoring districts, each “extra” student with a disability in the district sector would have to cost an additional $214,376 above the cost of educating a student in general education.  It is simply implausible that the slight gap in special education enrollments explains the substantial gap in funding between district and charter schools in New York City.

Like rookie sensation Aaron Judge, this report has lots of hits besides just the homeruns described above, so check it out.  In sum, New York City has made a major commitment to provide material support to students in its public charter schools.  Still, inexplicable funding inequities persist depending simply on whether a child is in a charter or a district school.  Larry and I think this case study provides yet another Reason to support weighted student funding with full portability (see what I did there?).  Switching to such a simple and equitable method for funding all public school students definitely would put us in a “New York State of Mind.”

If You Mostly Care About Test Scores, Private School Choice Is Not For You

April 28, 2017

If you mostly care about test scores, private school choice is not for you.  Despite the vast majority of randomized control trials (RCTs) of private school choice showing significant, positive test score effects for at least some subgroups of students, some of those gains have been modest and other effects have been null for at least some subgroups.  And now we have two RCTs, in Louisiana and DC, showing  significant test score declines for at least some subgroups and in some subjects.  The Louisiana decline is large and across-the-board, but the significant, negative effect in the new DC study appears to be  “driven entirely by students in elementary grades not previously in a needs-improvement school.

People will quibble over why these new DC results showed at least a partial decline.  They will note that the prior RCT of DC vouchers showed significant test score gains after three years (although the p value rose to .06 in year four even as the positive estimate remained).  They will note that vouchers in DC are worth almost 1/3 as much as the per pupil funding received by DC’s traditional public schools and almost half as much as DC’s charter schools.  Imagine how they might do if they received comparable resources (and yes resources can matter if there are proper incentives to use resources productively).  They will note that almost half of the control group attended charter schools, so to a large degree this study is a comparison of how students do in vouchers relative to charters.

But these largely miss the point — the benefits of private school choice are clearly evident in long term outcomes, not near-term test scores.  In the same DC program that just produced disappointing test score effects, using a voucher raised high school graduation rates by 21 percentage points.  Similarly, private school choice programs in Milwaukee and New York City were less impressive in their test score effects than in later educational attainment, where private school students in both cities were significantly more likely to enroll in college.

But if what you really care about is raising test scores, you’d be pushing no-excuses charter schools.  Rigorous evaluations, like the one in Boston, show huge test score gains for students randomly assigned to no-excuses charter schools.  You don’t even have to have school choice to produce these gains.  The same team of researchers showed that schools converted into no-excuses charters as part of a turnaround effort produced similarly big gains for students who were already there and did nothing to choose it.  The lesson that a fair number of foundations and policymakers draw is that we don’t need this messy and controversial choice stuff.  They believe that they have discovered the correct school model — it’s a no excuses charter — and all we need to do is get as many disadvantaged kids into these kinds of schools as we can, with or without them choosing it.

Unfortunately, no excuses charters don’t seem to produce long-term benefits that are commensurate with their huge test score gains.  The Boston no excuses charter study, for example, shows no increase in high school graduation rates and no increase in post-secondary enrollment despite large increases in test scores.  It’s true that students from those schools who did enroll in post-secondary schooling were more likely to go to a 4 than 2 year college, but it is unclear if this is a desirable outcome given that it may be a mismatch for their needs and this more nuanced effect is not commensurate with the giant test score gains.

This same disconnect between test scores and later life outcomes exists in several rigorously conducted studies of charter schools, including those of  the Harlem Promise Academy, KIPP, High Tech High, SEED boarding charter schools, and no excuses charters in Texas.  While of course we would generally like to see both test score gains and improved later life outcomes, the thing we really care about is the later life outcomes.  And the near-term test scores appear not to be very good proxies for later life outcomes.

So, what should we think about these new test results from DC vouchers, showing some declines for students after one year in the program?  We already know from rigorous research that the program improves later life outcomes, so I don’t think we should be particularly troubled by these test results.  It may be that control group students are in schools that will fare as well or better on test score measures.  But we should remember that 42% of that control group are in the types of charter schools that other research has shown can produce giant test score gains without yielding much in later life outcomes.  And we know that treatment group students are in a program that has previously demonstrated large advantages in later life outcomes.

I understand that many reporters, foundations, and policymakers act like they mostly care about test scores and these new results from DC have them all aflutter.  But if people could only step back for a second and consider what we are really trying to accomplish in education, the evidence is clearly supportive of private school choice in DC and elsewhere.

(edited to correct error noted in comments)

Evidence for the Disconnect Between Changing Test Scores and Changing Later Life Outcomes

November 5, 2016

Over the last few years I have developed a deeper skepticism about the reliability of relying on test scores for accountability purposes.  I think tests have very limited potential in guiding distant policymakers, regulators, portfolio managers, foundation officials, and other policy elites in identifying with confidence which schools are good or bad, ought to be opened, expanded, or closed, and which programs are working or failing.  The problem, as I’ve pointed out in several pieces now, is that in using tests for these purposes we are assuming that if we can change test scores, we will change later outcomes in life.  We don’t really care about test scores per se, we care about them because we think they are near-term proxies for later life outcomes that we really do care about — like graduating from high school, going to college, getting a job, earning a good living, staying out of jail, etc…

But what if changing test scores does not regularly correspond with changing life outcomes?  What if schools can do things to change scores without actually changing lives?  What evidence do we actually have to support the assumption that changing test scores is a reliable indicator of changing later life outcomes?

This concern is similar to issues that have arisen in other fields about the reliability of near-term indicators as proxies for later life outcomes.  For example, as one of my colleagues noted to me, there are medicines that are able to lower cholesterol levels but do not reduce — or even may increase — mortality from heart disease.  It’s important that we think carefully about whether we are making the same type of mistake in education.

If increasing test scores is a good indicator of improving later life outcomes, we should see roughly the same direction and magnitude in changes of scores and later outcomes in most rigorously identified studies.  We do not.  I’m not saying we never see a connection between changing test scores and changing later life outcomes (e.g. Chetty, et al); I’m just saying that we do not regularly see that relationship.  For an indicator to be reliable, it should yield accurate predictions nearly all, or at least most, of the time.

To illustrate the un-reliability of test score changes, I’m going to focus on rigorously identified research on school choice programs where we have later life outcomes.  We could find plenty of examples of disconnect from other policy interventions, such as pre-school programs, but I am focusing on school choice because I know this literature best.  The fact that we can find a disconnect between test score changes and later life outcomes in any literature, let alone in several, should undermine our confidence in test scores as a reliable indicator.

I should also emphasize that by looking at rigorous research I am rigging things in favor of test scores.  If we explored the most common use of test scores — examining the level of proficiency — there are no credible researchers who believe that is a reliable indicator of school or program quality.  Even measures of growth in test scores or VAM are not rigorously identified indicators of school or program quality as they do not reveal what the growth would have been in the absence of that school or program.  So, I think almost every credible researcher would agree that the vast majority of ways in which test scores are used by policymakers, regulators, portfolio managers, foundation officials, and other policy elites cannot be reliable indicators of the ability of schools or programs to improve later life outcomes.

With the evidence below I am exploring the largely imaginary scenario in which test scores changes can be attributed to schools or programs with confidence.  Even then, the direction and magnitude of changing test scores does not regularly correspond with changing later life outcomes.  I’ve identified 10 rigorously designed studies of charter and private school choice programs with later life outcomes.  I’ve listed them below with a brief description of their findings and hyperlinks so you can read the results for yourself.

Notice any patterns? Other than the general disconnect between test scores and later life outcomes (in both directions), I notice that the No Excuses charter model that is currently the darling of the ed reform movement and that New York Times columnists have declared as the only type of “Schools that Work” tend not to fare nearly as well in later outcomes as they do on test scores.  Meanwhile the unfashionable private choice schools and Mom and Pop charters seem to do much better on later life outcomes than at changing test scores.  I don’t highlight this pattern as proof that we should shy away from No Excuses charters.  I only mention it to suggest ways in which over-relying on test scores and declaring with confidence that we know what works and what doesn’t can lead to big policy mistakes.

Here are the 10 studies:

  1. Boston charters (Angrist, et al, 2014) – Huge test score gains, no increase in HS grad rate or postsecondary attendance. Shift from 2 to 4 yr
  2. Harlem Promise Academy (Dobbie and Fryer, 2014) – Same as Boston charters
  3. KIPP (Tuttle, et al, 2015) – Large test score gains, no or small effect on HS grad rate, depending on analysis used
  4. High Tech High (Beauregard, 2015) – Widely praised for improving test scores, no increase in college enrollment
  5. SEED Boarding Charter (Unterman, et al, 2016) – same as Boston charters
  6. TX No Excuses charters (Dobbie and Fryer, 2016) – Increase test scores and college enrollment, but no effect on earnings
  7. Florida charters (Booker, et al, 2014) – No test score gains but large increase in HS grad rate, college attendance, and earnings
  8. DC vouchers (Wolf, et al, 2013) – Little or no test score gain but large increase in HS grad rate
  9. Milwaukee vouchers (Cowen, et al, 2013) – same as DC
  10. New York vouchers (Chingos and Peterson, 2013) – modest test score gain, larger college enrollment improvement

The Over-Confidence of Portfolio Management

October 25, 2015

I’ve been having a debate over the last few weeks with Neerav Kingland and others about the dangers of a high-regulation approach to school choice.  (You can see my posts on this so far here, here, here, here, here, and here).  I know this seems like a lot of posts on a topic, but as one grad student observed, it took more than 100 pieces about the obvious error of government reported high school graduation rates before people fully acknowledged the error and too significant steps to correct it.  Let’s hope convincing ed reform foundations and advocates to scale way back on their infatuation with heavy regulation does not require the same effort as moving more obstinate and dim-witted government officials.

One heavy-handed regulatory approach that is particularly worrisome is the strategy of portfolio management.  Under this strategy, a portfolio manager, harbor master, or some other regulator actively manages the set of school options that are available to families by closing those believed to be sub-par and expanding or replicating those that are thought to be more effective.  This approach is being implemented in New Orleans and the city appears to be experiencing significant gains in achievement tests, so Neerav and others are puzzled as to why I don’t support it.

I’ve tried to express my reasons for opposing portfolio management in several ways.  I tried mocking it: “If education reform could be accomplished simply by identifying and closing bad schools while expanding good ones, everything could be fixed already without any need for school choice.  We would just issue regulations to forbid bad schools and to mandate good ones.  See?  Problem solved.”  That clearly didn’t work because folks like NOLA advocate Josh McCarty replied: “moving the left end of the performance curve to the right through regs has gotten more kids in higher perf schools.”

So, let me try again.  A portfolio manager can only move “the left end of the performance curve” if the regulator can reliably identify which schools are likely to harm students’ long-term outcomes and which ones are likely to improve them.  If you don’t really know whether schools are on the left or right end of some curve of quality, closing schools just limits options without improving long-term outcomes.  But backers of portfolio management are not lacking in confidence.  They have achievement test results, so they think they know which are the good and bad schools.

Unfortunately, they are suffering from over-confidence.  Achievement tests are useful but they are not nearly strong enough predictors of later life outcomes to empower a portfolio manager to close a significant number of schools because he or she “knows” that those schools are “bad.”  In fact, the research I reviewed on rigorous evaluations of long-term outcomes from choice programs suggests that using test scores to decide whether a bunch of schools should be closed or expanded would lead to significant Type 1 and Type 2 errors.  That is, in their effort to close bad schools, portfolio managers may very well close schools with lower test performance that actually improve high school graduation, college-attendance, and lifetime earnings.  And they may expand or replicate schools that have high test performance but do little to improve these later life outcomes.

If there were an active portfolio manager of Florida charter schools, they would have closed a bunch of charter schools that were doing a great job of improving students’ later life outcomes.  As Booker, et al’s research shows, relying solely on test scores to distinguish good from bad schools would lead to serious errors by an active portfolio manager:

The substantial positive impacts of charter high schools on attainment and earnings are especially striking, given that charter schools in the same jurisdictions have not been shown to have large positive impacts on students’ test scores (Sass, 2006; Zimmer et al., 2012)…. Positive impacts on long-term attainment outcomes and earnings are, of course, more consequential than outcomes on test scores in school. It is possible that charter schools’ full long-term impacts on their students have been underestimated by studies that examine only test scores. More broadly, the findings suggest that the research examining the efficacy of educational programs should examine a broader array of outcomes than just student achievement. (pp. 27-8)

Conversely, foundations and portfolio managers are pouring more resources into certain types of schools with strong test performance that are failing to show much benefit for students’ long-term outcomes.  As Angirst, et al, Dobbie and Fryer, and Tuttle, et al show, a bunch of charter schools with large achievement test gains, including Boston “no-excuses” schools, Harlem Promise Academy, and KIPP, have produced little or nothing in terms of high school graduation and college-attendance rates.

Portfolio management guided solely by test scores would seriously harm students by unwittingly closing a bunch of successful schools, like those Booker, et al studied in Florida, while expanding and pouring more resources into ones with less impressive long-term results, like those studied by Angirst, et al, Dobbie and Fryer, and Tuttle, et al.

Matt Barnum challenged me on Twitter to describe what evidence would persuade me to support portfolio management.  At a minimum I would want to see that portfolio managers have reliable tools for predicting long-term outcomes for students so they knew which choice schools should be closed and which should be expanded or replicated.  The evidence I’ve reviewed here and in more detail in this prior post suggests that they do not have a reliable tool and so the entire theory of portfolio management falls apart. I’m not making the strawman argument that test scores are useless or that no school should ever be closed by regulators.  I’m just arguing that portfolio management requires confidence in the predictive power of achievement tests that is not even close to being warranted by the evidence.

But what about the impressive achievement gains that Doug Harris and his colleagues find are being produced in New Orleans?  Let’s keep in mind that many reforms have been implemented in New Orleans at the same time.  Even if we were confident that the test score gains in New Orleans are not being driven by changes in the student population following Katrina (and Doug and his colleagues are doing their best with constrained data and research design to show that), and even if these test score gains translate into higher high school graduation and college attendance rates (which Doug and his colleagues have not yet been able to examine), we still would have no idea whether portfolio management and other high regulations in NOLA helped, hurt, or made no difference in producing these results.  In fact, the evidence from the 7 rigorous studies on school choice programs with long-term outcomes suggests that portfolio management and other heavy regulations are neither necessary nor desirable for producing long-term gains for students.

Neerav, Matt Barnum, and Josh McCarty have suggested that I am making overly-broad claims not consistent with evidence.  I think the opposite is true.  I’ve carefully cited and quoted the relevant research and drawn the obvious conclusion — active portfolio management based on achievement tests is likely to make harmful errors and unnecessarily restrict options.  In fact, it seems to me that the burden is on supporters of portfolio management to demonstrate that they are able to reliably distinguish between schools with good and bad long-term outcomes.  If you are going to go around telling families that they can’t choose a certain school because it is bad for them, you had darn better be confident that it really is bad.

Does regulation improve the political prospects for choice?

October 6, 2015

In this series of post against the high-regulation approach to school choice, I have demonstrated that performance accountability is not typical of government programs and that heavy regulation drives away quality supply, hurting rather than protecting the students these regulations are meant to help.  If high-regulation is not the norm and does not help children, supporters of this approach might still favor it if they think it has certain political advantages.

For those interested in private school choice, two political advantages are claimed: 1) High-regulation addresses some  objections, winning votes among skeptics to improve the political prospects of passing and sustaining those programs; 2) High-regulation protects private school choice programs from the political damage caused by scandals and embarrassing outcomes.

Neither of these arguments is supported by experience.  Conceding regulatory measures to skeptics and opponents has hardly changed a single vote.  Backers of the Milwaukee voucher program thought they would get relief from legislative opposition if they accepted more burdensome regulation.  No votes have changed as a result and the program remains as precarious as ever.  Nor has regulation protected programs from scandal.  Judging from the steady stream of news reports about teachers in traditional public schools sleeping with students, it appears that no amount of background checks or government oversight can eliminate rare but regular instances of misconduct.  I’m not arguing against a reasonable and light regulatory framework, I’m just suggesting that higher levels of regulation provide little or no additional political protection.  Determined opponents can always find scandals to exploit and cannot be appeased with anything short of preserving the traditional public system.

I’m actually more worried that key backers of school choice are starting to abandon private school choice and focus all of their energies on charters.  High-regulation is the norm in charter programs.  You don’t have to worry about charter schools refusing to participate in a heavily regulated program since they have no alternatives.  And charters seem to be flourishing.  Charter programs exist in more states with more schools serving more students than do private choice programs.  Many important backers of school choice seem to believe that charters are also getting better results.  As Neerav Kingsland of the Arnold Foundation tweeted yesterday: “why is it the over-regulated charter sector that has had the most breakthroughs with low income students?”

Unfortunately, Neerav is mistaken.  Charters are not producing better results than private school choice.  High-regulation comes with a cost to quality.  Let’s consider rigorous evidence on how charter and private school choice affect educational attainment.  For reasons I will discuss at greater length in the next post, I think attainment is a more meaningful indicator of long-term benefits than achievement test results.  I’m aware of 4 rigorous studies of the effect of charter schools on attainment.  The general pattern among them is that programs producing large gains in achievement test outcomes are producing little or no increase in educational attainment.

Angrist, et al examined Boston charter schools and found significant benefits for charter students on MCAS, SAT, and AP performance.  On attainment they write:

Does charter attendance also increase high school graduation rates? Perhaps surprisingly given the gains in test score graduation requirements reported in Table 4, the estimates in Table 7 suggest not. In fact, charter attendance reduces the likelihood a student graduates on time by 12.5 percentage points, a statistically significant effect. This negative estimate falls to zero when the outcome is graduation within five years of 9th-grade entry. (p. 15)

Nor are results much better for attending college: “While the estimated effect of charter attendance on college attendance is positive, it is not large enough to generate a statistically significant finding.” (p. 16)  Angrist, et al do find a significant shift of students from attending 2 year to 4 year colleges, but we don’t know yet if that shift represents a positive development until we see whether they complete their degrees.  Shifting students to 4 year college for which they are ill-suited and from which they drop out does them no favor.

Dobbie and Fryer examine the results of a single charter school in Harlem, the Promise Academy.  Like Angrist, et al, they find large achievement test gains but little benefit for attainment.  Dobbie and Fryer find a higher high school graduation rate after 4 years of the start of 9th grade, but it disappears by 6 years. (p, 18)  College attendance benefits are also fleeting: “Similar to the results for high school graduation,however, control students eventually catch up and make the treatment effects on college enrollment insignificant.”  Dobbie and Fryer similarly find a shift toward 4 year colleges, but again this result is ambiguous. Four year college should help students obtain more schooling but they report “The number of total semesters enrolled in college between lottery winners and lottery losers is small and statistically insignificant.” (p. 19)

Tuttle, et al’s recent evaluation of KIPP charter schools also finds large achievement test gains for charter students but little or no attainment benefit.  Tuttle and her team at Mathematica make two types of comparisons to assess the progress of KIPP high school students.  In one they find: “For new entrants to KIPP high schools, we also examine the probability of graduating within four years of entry. We find that this group of KIPP high schools did not significantly affect four-year graduation rates among new entrants.” (p. 36)  When they examine students who continued from KIPP middle schools into KIPP high schools, they find a small but statistically significant drop in the rate at which students drop out — about 2 percentage points. (p. 39)

Booker, et al examine charter schools in Chicago and Florida and find significant benefits in educational attainment as well as higher earnings later in the workforce — at least for Florida charter students.  They write: “In Florida, the charter high school students show a consistent advantage in absolute terms of 8 to 11 percentage points from high school graduation through a second year of college enrollment.” (p. 22)  On later earnings they find: “Charter high school attendance is
associated with an increase in maximum annual earnings for students between ages 23 and 25 of $2,347—or about 12.7 percent higher earnings than for comparable students who attended a charter middle school but matriculated to a traditional high school.”

Before the high-regulation folks get too excited about the Booker, et al results as vindication of their approach, they should note that these charter schools did not produce impressive achievement test results.  Booker, et al write:

The substantial positive impacts of charter high schools on attainment and earnings are especially striking, given that charter schools in the same jurisdictions have not been shown to have large positive impacts on students’ test scores (Sass, 2006; Zimmer et al., 2012)…. Positive impacts on long-term attainment outcomes and earnings are, of course, more consequential than outcomes on test scores in school. It is possible that charter schools’ full long-term impacts on their students have been underestimated by studies that examine only test scores. More broadly, the findings suggest that the research examining the efficacy of educational programs should examine a broader array of outcomes than just student achievement. (pp. 27-8)

In the high-regulation approach, these charter schools might well be identified as the “bad” schools for failing to improve test scores, and yet they are the ones that produce long-term success for their students.  In the high-regulation approach a portfolio manager or harbor master might kick these schools out of the program or restrict their growth for failing to produce achievement gains.

Let’s briefly review the results from the three rigorous examinations of the effect of private school choice on educational attainment.  Unlike the charter research, they all show significant benefits for attainment.  Wolf, et al examined the federally funded DC voucher program.  They found little benefit for voucher students on achievement tests but those students enjoyed a 21 percentage point increase in the rate at which they graduated high school.  Cowen, et al examined the public funded voucher program in Milwaukee and found a 5 to 7 percentage point increase in the rate at which voucher students attended college.  And Peterson and Chingos examined a privately funded voucher program in New York City and found that African-American voucher recipients experienced a 9 percentage point increase in attending college.  There was no significant benefit for Hispanic students.

If the high-regulation folks wanted to ditch private school choice to go all-in on charters, they would be making a horrible mistake.  The evidence suggests private school choice is producing stronger long-term results.  In addition, among charter schools, the kinds of schools that high-regulation folks like the most are the ones producing weaker long-term outcomes.  Focusing only on charters making the biggest achievement score gains would miss those charters with more modest achievement results but truly impressive attainment outcomes.  Charter schools offer the illusion of getting the benefits from choice without too much of the messiness markets.  As it turns out, central planning among charter schools is no easier than central planning among traditional public schools.

In addition to losing quality if key choice backers were to support charters to the exclusion of private school choice, there are obvious political advantages to backing both types of choice.  Private school choice has helped make the world safe for charters by taking more of the political heat.  We wouldn’t have the same expanding charter sector were it not for the credible threat of even more private school choice.  And the choice movement would be wise to spread its bets across a variety of approaches to expanding school choice.  No one knows the ideal political strategy or regulatory scheme, so having a variety of different approaches allows us to learn about how these different methods for expanding choice are doing.  We need choice among choice.

%d bloggers like this: