
Check out my two-part interview on education philanthropy with Mike Hartmann: Part 1 Part 2
Readers of JPGB will recognize many themes from earlier posts like:
Advice to the Arnold Foundation
Political Science for Ed Reform Dummies

Check out my two-part interview on education philanthropy with Mike Hartmann: Part 1 Part 2
Readers of JPGB will recognize many themes from earlier posts like:
Advice to the Arnold Foundation
Political Science for Ed Reform Dummies

Josh McGee and I have an oped in the Houston Chronicle today arguing against state takeovers. Here’s the money quote:
State policymakers may imagine that they are smarter and better than the elected officials they would displace, but, even if they were right, the intelligence and goodness of the school board is hardly the issue. Distant and unaccountable bureaucrats, no matter how well-trained and well-intentioned, are unlikely to understand and address the specific needs of communities as well as locally elected officials are, no matter how fractious and chaotic they may appear. Conservatives have long understood this principle, which is why they have traditionally supported decentralization of responsibility over schools to local governments, communities, and families, so it is puzzling that self-styled conservatives in Texas would support state takeovers.
There is no simple solution to chronic low academic performance, but the problem is almost certainly better addressed by empowering communities and families rather than disenfranchising them.
This is a mistake reformers have made over and over. It’s time we learn from our mistakes and avoid repeating them.

Nudge interventions, in which students receive texts encouraging them to do things that are thought to be good for them, have yielded another disappointing result. In a newly released study, Kelli A. Bird, Benjamin L. Castleman, Jeffrey T. Denning, Joshua Goodman, Cait Lamberton, and Kelly Ochs Rosinger report:
We investigate, through two randomized controlled trials, the impact of a national and state-level campaign to encourage students to apply for financial aid for college. The campaigns collectively reached over 800,000 students, with multiple treatment arms to investigate different potential mechanisms. We find no impacts on financial aid receipt or college enrollment overall or for any student subgroups. We find no evidence that different approaches to message framing, delivery, or timing, or access to one-on-one advising affected campaign efficacy.
For those keeping tabs, I expressed my strong doubts about texting in this review of Ben Castleman’s book even as Bill Gates was praising texting interventions and NPR’s Hidden Brain was featuring it as a success. And then my skepticism was strengthened when texting resulted in null to negative college completion outcomes in a study that received little attention. Now even its early proponents are finding disappointing results.
But this isn’t the first time I’ve predicted that a reform initiative would fail when it would take foundations years and millions of dollars before reaching the same effective conclusion. I was raising alarms about the Measuring Effective Teacher (MET) initiative while almost everyone was jumping on the teacher evaluation and quality bandwagon. Several years later and without much fanfare we finally hear that the initiative yielded virtually no benefits. Similarly, when a large chunk of reformers were getting behind no-excuse charter schools as the correct school model, I was warning that the model appeared unable to yield longer-term benefits, so we might not want to put all of our eggs (and all students) into the no-excuse basket. And recently, we’ve seen that Boston no-excuse charter schools have no effect on students completing college. Lastly, I’ve been predicting the political and educational failure of Portfolio Management for some time now. While the Arnold Foundation has doubled down by joining Reed Hastings in giving $200 million to the City Fund to push the idea, I expect it will be shortly after they burn through that money that we’ll hear about a reconsideration of their reform strategy.
Why am I able to anticipate these failures in education reform initiatives, while the people devoting fortunes to these efforts and their staff have such a hard time avoiding strategies that result in failure? I’m not that smart and they aren’t that dumb. I suspect the answer is that foundations have organizational interests that tend to draw them to a mistaken theory about education policy. In its essence, that theory holds that there are policy interventions that could improve outcomes for large numbers of students if only we could discover them and get policymakers and practitioners to adopt them at scale.
I begin with a very different theory. I suspect that there are relatively few educational practices that would produce uniformly positive results. Instead, I’m inclined to think of education as similar to parenting, in which the correct approaches are highly context-specific. Even within the same family, we may choose to parent different children facing similar issues in very different ways. There may be some uniformly desirable parenting practices, but most of them are already known and widely disseminated. So, if we wanted to improve parenting, the best we could do would be to empower parents to be in a better position to judge their context and make their own decisions about how to raise their children. Similarly, the best we could do to improve education is to empower families and communities to make decisions within their own context. There is relatively little we could tell all schools or educators to do to improve outcomes.
But foundations and the research community that follows their lead have a very hard time with this kind of theory. They think their job is to use science to identify the correct educational practices and then get everyone on board with doing it. To decentralize solutions to communities and families is to relinquish control and the status of expertise. So, despite repeated failures in finding top-down policies and practices to improve outcomes at scale, they continue to search for what can’t be found.
As Mike McShane and I pointed out in our edited volume, there is nothing inherently wrong with failure in education policy. The problem is if we refuse to acknowledge that failure and learn from it. Given the string of reform failures that large foundations have experienced over the past decade, it might be productive if they devoted some serious energy to reconsidering their basic assumptions about top-down policy solutions and became more open to the possibility that the most they could do is to empower communities and families to find their own solutions.

Josh Angrist, the MIT economist and a leading voice on research methods and education policy, has a recent piece in Forbes in which he praises the dawning of a new era in which policymakers are guided by economists conducting experimental analyses of promising education reforms. He writes:
Alas, school reform has rarely been grounded in the sort of empirical analysis required of a new drug or medical treatment. Many educational innovations are propelled primarily by a politician or philanthropist’s good feelings. It shouldn’t surprise us that weakly researched innovations often lead to disappointing results. But this unscientific approach is now changing. America’s large urban districts are piloting new models for education delivery, such as small schools, charter schools, various sorts of magnet programs, and vouchers. Importantly, these innovations are often deployed through experiments… Economists nowadays use these experiments to provide credible, non-partisan evidence on the consequences of school reform.
To be sure, experimental methods are the best way to identify causal effects, and most of my own research uses this approach. Unfortunately, this improvement in methods does not always yield credible and non-partisan evidence because it is all too common for researchers to misinterpret the policy implications of these experiments, even when they are properly conducted. Several examples of this type of misinterpretation can be found in Angrist’s brief Forbes article. I’ll pick one to illustrate the point.
One of Angrist’s claims is that a certain type of charter school has been demonstrated as an effective policy with this rigorous new approach to research: “I’ve seen compelling evidence that urban charter schools emphasizing high expectations and data-driven instruction are winners, capable of closing the black-white achievement gap in just a few years.” The evidence to which Angrist is primarily referring is the experimental evaluation of Boston charter schools in which he has been involved with several co-authors. That research has shown large test score gains among students admitted to those Boston charters by lottery relative to those not admitted.
The problem is that increasing test scores does not necessarily mean that a policy is a “winner.” Test scores are an imperfect proxy for a set of knowledge and skills that we hope translate into greater educational and life success for students. Unfortunately, a growing body of research is showing a disconnect between changing test scores and changing later life outcomes for students. But we don’t have to look across the entire research literature to find numerous examples of this disconnect between changing test scores and changing later life outcomes. We can find evidence of it in the very Boston charter schools on which Angrist relies for his claim.
A new study by one of Angrist’s former students, Elizabeth Setren, examined test scores for students admitted by lottery to Boston charters but also tracked those students all the way through college completion. The main purpose of her study was to disaggregate effects for special needs and English language learner (ELL) students, so she never actually reports the combined results for all students. But we can see from the results for general education students, who comprise the vast majority of students in the study, what the overall results must be.
Like Angrist’s previous research, Setren finds large test score gains for students admitted to Boston charter schools by lottery. As shown in Table 4, general education students admitted to Boston charters benefit by .268 standard deviations (sd) on math tests and .163 sd on English Language Arts tests. ELL and special ed charter students show similar test score benefits. But as shown in Table 5, Boston charter school students are no more likely to graduate from high school than the lotteried control group, even five or six years after starting high school.
In Table 6, we can see that despite this lack of improvement in high school graduation rates, Boston charters are more likely to have their general education students enroll in post-secondary education, driven largely by an increase in enrollment in 4-year institutions with a possible decline in enrollment in 2-year schools. Boston charters’ special needs students show no statistically significant increase in post-secondary enrollment. Toward the bottom of Table 6 we can see college completion rates. Neither special needs nor general education students are more likely to complete a post-secondary degree in 4 years than the control group of students denied admission to Boston charters by lottery. In fact, the estimated effect for general education students is negative, but not statistically significant.
So, the overall picture does not show a policy that is a “winner.” One of Angrist’s former students, using the type of experimental method he endorses to examine the policy he claims is proven to work actually shows that in the long run the policy may produce no benefits or may even produce a harm. General education students admitted by lottery to Boston charters do experience large test score benefits, but they are no more likely to graduate high school. Those students are also more likely to enroll in post-secondary education but no more likely to obtain a post-secondary credential than the control group. Students who take out loans to enroll in college but do not finish it may be worse off, so this pattern of results may suggest that Boston charters actually harm their students’ long-term educational outcomes.
And once again large gains in test scores are not a reliable proxy for improvement in later life outcomes. In the Forbes piece, Angrist suggests otherwise: “Though imperfect, test-based measures of value-added predict gains in important economic outcomes like college enrollment and earnings.” Notice the rhetorical sleight of hand in Angrist’s claim. The issue is not whether test scores are correlated with later life outcomes but whether rigorously identified changes in test scores produced by policy interventions translate into later changes in life outcomes. In the case of Boston charters, changes in test scores are not consistent with changes in later life outcomes, at least for general education students who constitute the bulk of the program.
Angrist is right that experiments are good and useful. But he is wrong about the dawning of a new age of science-driven education policymaking. Science is only as good as the proxies we use for outcomes we may really care about and only as reliable as the accuracy with which the scientists describe the research literature. So, when economists come to policymakers to say that science has spoken and we now know what works, policymakers have every reason to retain some skepticism.
Updated post note — The original version of this post noted that special needs students failed to improve on test scores but did show a higher likelihood of completing a 2 year college. That was a correct reading of the results as displayed in Table 4, but was inconsistent with the text of the paper. The author has acknowledged that the table was in error, so I have modified the post to reflect her corrected results.
My favorite part of this is the invocation of “data” to prove the click-bait opinion that the US is just OK. Technocratic and anti-patriotic is precisely the NYT brand.
Of course, the most relevant data might be net migration (or attempted net-migration). After all, unlike the Soviet Empire, no one is proposing a wall to keep people in. But the beauty of technocracy is that the technocrats get to pick the metrics.
Similarly, when it comes to school choice one might think that economists would be persuaded simply by the fact that people choose schools to believe that those are likely better for them — revealed preference. But no. They demand test scores, integration measures, social-emotional learning scales, etc.. to judge chosen school quality. Keep measuring (more likely mis-measuring) until the technocrat can find the metric to show how your own better judgement is mistaken.

(Guest post by Patrick J. Wolf)
I’ve led or assisted with seven rigorous longitudinal evaluations of privately- or publicly-funded private school choice programs. Each one has yielded a big surprise.
The three-city evaluation Paul Peterson led discovered that partial-tuition K-12 scholarships had no clear effect on student test scores overall, but clearly benefited African American students. The original evaluation of the DC Opportunity Scholarship Program that I led for the U.S. Department of Education found only suggestive evidence of achievement effects of the federal school voucher program, and only in reading, but identified big positive effects of the program on high school graduation rates. The Milwaukee Parental Choice Program evaluation I co-led with John Witte confirmed that the original urban school voucher program had clear effects on multiple measures of student educational attainment but also produced reading test score gains when the tests were high-stakes for the private schools in the program. An experimental study of a partial-tuition scholarship program in India, for which I provided assistance, found clear achievement effects only for girls.
Then there is our four-year longitudinal evaluation of the Louisiana Scholarship Program. In our first set of reports we found that the LSP had positive effects on public school racial integration, as reported here, here, and here. We also discovered that competitive pressure from the statewide expansion of the school voucher program in 2012 had neutral or positive effects on the test scores of students in affected public schools. Our big surprise, however, was that participants in the voucher program scored significantly lower on the state accountability test than their control group peers, especially in the first year of the program and particularly in math. Those negative test score effects of the voucher program were somewhat smaller after two years and even statistically insignificant in Year 3, when the state switched to a different test and held schools harmless for the results. What would happen in Year 4? More surprises, it turns out.
Today we released the results of our final set of four technical research reports on the LSP. The experimental impacts of winning a lottery to your first-choice private school and enrolling in that school for any period of time were negative and back to statistically significant for all of our statistical models in math and some of them in reading. African American students experienced smaller negative achievement effects than did students of other races. Students whose first-choice private schools had higher tuitions, larger enrollments, and longer school days experienced relatively “better” test score effects than students whose first-choice schools didn’t have those features.
Winning an LSP school lottery had no impact on the rates at which students in our study subsequently enrolled in either a two-year or four-year college. The rate was 60.0% for students in the LSP experimental treatment group and 59.5% in the experimental control group. Those college enrollment rates are relatively high for the population of low-income students eligible to apply for the LSP, in part because Louisiana has enacted a series of programs to encourage college access.
Finally, the LSP succeeded in attracting students from economically and socially disadvantaged backgrounds. The students who remained in the LSP three years after applying to it were more likely to be eligible for the federal lunch program, African American, and female than the average K-12 student in Louisiana. Students with lower initial test scores than the state average more likely to apply to the LSP than were students with higher test scores. Among the students who won LSP school lotteries, those with lower initial test scores were more likely to use an LSP voucher for at least three years.
Debates will rage about what has been learned from this one, latest, rigorous evaluation of a private school choice program. Commentators should keep in mind that the LSP has design features that make it unlike most school choice programs. It is the only statewide school voucher program that requires participating schools to adopt an open admissions policy and administer the state accountability test to their voucher-using students. Survey experiments here, and here, have found that those two regulatory provisions tend to decrease the interest of private school leaders in participating in school choice programs. Only about one-third of Louisiana private schools choose to participate in the LSP.
Serious, rigorous studies of private school choice programs should continue. So far, results keep surprising because it appears that the effects of these programs are highly dependent on their design and context. I wouldn’t be surprised if more surprises await.

It is time once again to (dis)honor the recipient of the William Higinbotham Inhumanitarian Award. We have a smaller but still (un)excellent set of nominees to consider. I failed to submit my own nominee as I was paralyzed by such a target-rich environment and then came down with a fever just when I needed to make a decision and write it. Oh well. Higgy nominees are evergreen, so I’ll keep those possibilities in mind for next year.
So we had three nominees to consider: Richard Henry Pratt, nominated by Matt, Kosoko Jackson, nominated by Greg, and William N. Sheats, nominated by Patrick Gibbons. While they are all very (un)worthy nominees, I think Kosoko Jackson is clearly most deserving. Pratt and Sheats were much more like BSDDers than the kind of PLDDers we are seeking for the Higgy.
Sheats mustered the coercive power of the state to amend Florida’s constitution to forbid integrated instruction in Florida public schools. Pratt embarked on a systematic government program to remove Native American children from their families to be educated in boarding schools that would raise them as “real Americans,” which was tantamount to obliterating an entire people, their language, their religion, and their customs. Whenever people start arguing that we need public schools to create a common sense of identity and shared understanding of democratic citizenship, remember that line of thinking ultimately leads to Pratt. Both Sheats and Pratt are horrifying, but too horrifying for the Higgy.
Jackson is the winning nominee because his dictatorial behavior was really about self-advancement, not truly oppressing large numbers of people. Jackson joined in social media witch hunts that falsely called out competing authors (falsely) for alleged infractions of the Young Adult Fiction politically correct code. That was working well to clear the path for his own career until the mobs came after Jackson.
Posturing on social media as woker-than-thou to advance one’s stupid career and at the expense of a commitment to truth, good sense, and professional courtesy… . Hmmm, does this sound like something familiar?
Anyhoo, Jackson joins past winners, John Wiley Bryant, Plato, Chris Christie, Jonathan Gruber, Paul G. Kirk, and the inaugural winner, Pascal Monnet.

(Guest Post by Patrick Gibbons)
William N. Sheats was Florida’s very first elected superintendent of public instruction, serving from 1893 to 1904 and again from 1913 until his death in 1922.
As the state’s leading educator, Sheats worked tirelessly to modernize Florida’s education system. He drafted the first statewide curriculum, reformed teacher training and required teachers to pass exams to prove subject-area mastery. He passed the state’s compulsory-attendance law in 1919 and made Florida’s public school system one of the best funded among southern states.
Contemporaries called him the “father of Florida’s public school system.”
As a chair of the education committee during Constitutional Convention of 1885, Sheats nearly caused panic among Democrats when he proposed allowing taxes to support the creation of common schools for black students. Thirty-two members of his own party voted against him. He invited Booker T. Washington to speak before white educators and helped secure public funds for the American Missionary Association’s Fesseden Academy, a college and career preparatory school in rural Florida. Sheats even lost his re-election in 1903 after Florida Education Association Vice President Clementine Hampton, along with the editor of the Gainesville Sun, smeared him as a “Friend of the Negro.”
One can argue that black students may have been worse-off without Sheats as the state superintendent, but the Higgy is not about recognizing the worst or most evil human being. For all the good he may have tried to accomplish, Sheats’ paternalistic racism left lasting scars.
Education, Sheats believed, would “make the vast number of idle, absolutely worthless negroes industrious and self-supporting.”
Sheats enshrined segregation into Florida’s Constitution of 1885, personally writing Article XII, Section 12 which states:
“White and colored children shall not be taught in the same school, but impartial provision shall be made for both.”
It would remain in Florida’s constitution until 1968.
Sheats fervently believed in racial segregation. As he saw it, “any effort to enforce mixed education of the races would forever destroy the public school system in one swoop.”
Fearing teachers might sway impressionable young minds, Sheats used the power of his office to outlaw hiring teachers trained at racially integrated northern colleges. He also outlawed white teachers from educating black students in public schools. The combination of strict certification requirements, a ban on white teachers educating black students, and a rule preventing teachers trained at integrated colleges meant many public schools for black students with a shortage of teachers. By 1924, two years after his death, the rules he left in place meant only 1 out of every 4 black teachers were state certified.
When Sheats learned of Orange Park Normal & Industrial School, a racially integrated private school operating outside of Jacksonville, he lashed out, calling the school a “social and moral blotch,” and a “vile encroachment upon our social and moral system.” With the constitution mandating racial segregation only in public schools, Sheats lobbied the legislature to pass a bill outlawing whites from educating black and white students within the same building.
The New York Times reported in 1896 that the law “provided that it should be a penal offense” for any person or organization to run a school, public, private or parochial, “wherein white persons and negroes should be instructed or boarded within the same building or taught in the same class, or at the same time, by the same teacher.” Those found guilty could be fined between $150 and $500 or imprisoned for three to six months.
With the law passed, Sheats ordered the arrest of the school principal, three patrons, five teachers and even the local minister. In October 1896, a judge in the Fourth Judicial Circuit Court of Florida tossed out the law on a technicality and Sheats went to work trying to pass a new law. He wouldn’t succeed until after returning to office in 1913. Known as “Sheats Law” the state of Florida now banned “white persons from teaching Negroes in Negro schools.” The law also prohibited black teachers from educating white students.
Facing declining enrollment, attacks from the KKK, wariness over legal expenses and threats of arrest, Orange Park Normal & Industrial School closed its doors for good. After two decades, Sheats had finally succeeded in shuttering Florida’s first and only racially integrated school. A Catholic school under the leadership of 34-year old Bishop Michael Joseph Curley stepped up to fight the law. When Sheats asked Bishop Curley to remove white teachers from the school he refused and vowed to fight the law all the way to the Supreme Court.
Arrests wouldn’t happen until April 1916 when Gov. Park Trammell, at the instance of Sheats, ordered the arrest of three Catholic sisters who taught at St. Benedict the Moore School in St. Augustine, Fla. They were charged with “unlawfully teaching negroes in a negro school.” The arrest of the three sisters attracted national attention, but it also forced black private schools throughout the state to temporarily cease operations or risk arrest.
Fortunately, the case was resolved quickly. On May 20, 1916, Judge George Cooper Gibs ruled the “Sheats Law” unconstitutional, declaring,
“Has a white teacher any the less right to sell his services to negro pupils than a white doctor to negro patients, or a white lawyer to negro clients, or a white merchant as a right to sell his goods to negro customers, and vice versa?”
Sheats enshrined segregation into Florida’s constitution and fathered a system of uniform public schools that were ultimately a system of separate and unequal schools. From 1885 until well into the 1960s, Florida’s public-school system required separate attendance zones for white and black students, even if they lived in the same neighborhood. Segregation became “so entrenched that school superintendents were required to keep separately the books used in white and Negro schools.” Even the tax dollars used for white and black schools could not comingle.
Although he dared to fund education for black students when many contemporaries of his time would not, he ultimately created and enforced a regime of racial segregation and inequality that lasted 83 years. For that, he deserves the dishonor of the Higgy.
I’ve lost track of how many studies with rigorous causal identification find that improving test scores is not associated with improving later life outcomes, like graduating high school, attending and completing college, and getting a job and earning higher salaries. In the area of school choice alone we now have dozens of such studies, but you can also find this achievement-attainment disconnect in rigorous studies of pre-school and other interventions.
You can add to this list the latest study from Mathematica on the long-term effects of attending a charter middle school. Mathematica examined lotteries at a sample of charter middle schools around the nation, comparing outcomes for lottery winners to those of lottery losers. When the original results were completed almost a decade a ago, they found no overall effect on test scores. But when they disaggregated results, they found that urban charters had a significantly positive effect while suburban charters had the opposite result. This prompted leaders in the charter and ed reform worlds to conclude that we should focus our efforts on urban charters, since those were the ones that “worked.”
Now Mathematica has followed-up on those students to see whether they eventually attended and completed college. The new results show that attending a charter middle school has no effect on attending or completing college, just as it had no overall effect on test scores. But — and this is the important part for our discussion — they also concluded:
The success of an individual charter middle school in improving college outcomes was not related to its success in improving middle school achievement. The study schools that improved middle school achievement were not consistently more successful than others in boosting college enrollment and completion.
So, all of those technocratically-minded ed reformers who thought we should focus on urban charters because test scores showed they “worked” were guilty of mis-judging long-term success based on unreliable short-term measures. The evidence shows that changing test scores is not a particularly good indicator of schools that will improve their students’ lives.
City Fund, NACSA, and others who support portfolio management, harbor-mastering, quarterbacking, or whatever marketing term they are using nowadays are once again left trying to explain exactly how they intend to distinguish the “good” charter schools from the “bad” ones better than parents can. But take comfort, their political ineptitude matches their technocratic inclinations so that I expect they will burn through their $200 million without successfully installing and maintaining any multi-sector portfolio management systems. So children will be safe from their falsely-guided superior judgement.

It is time once gain to solicit nominations for the William Higinbotham Inhumanitarian Award. Below I reproduce portions of the first announcement of “The Higgy” in 2012, so you have an understanding of the historic significance and criteria for this dishonor.
——————————————————————–
As someone who was recognized in 2006 as Time Magazine’s Man of the Year, I know a lot about the importance of awards highlighting people of significant accomplishment. Here on JPGB we have the Al Copeland Humanitarian Award, but I’ve noticed that “The Al” only recognizes people of positive accomplishment. As Time Magazine has understood in naming Adolf Hitler, Joseph Stalin, and Ayatullah Khomeini as Persons of the Year, accomplishments can be negative as well as positive.
(Then again, Time has also recognized some amazing individuals as Person of the Year, including Endangered Earth, The Computer, Twenty-Five and Under, and The Peacemakers, so I’m not sure we should be paying so much attention to what a soon-to-be-defunct magazine does. But that’s a topic for another day when we want to talk about how schools are more likely to be named after manatees than George Washington.)
Where were we? Oh yes. It is important to recognize negative as well as positive accomplishment. So I introduce “The Higgy,” an award named after William Higinbotham, as the mirror award to our well-established “Al.”
Just as Al Copeland was not without serious flaws as a person, William Higinbotham was not without his virtues. Higinbotham did, after all develop the first video game. But Higinbotham dismissed the importance of that accomplishment and instead chose to be an arrogant jerk by claiming that his true accomplishment was in helping found the Federation of American Scientists and working for the nonproliferation of nuclear weapons. I highly doubt that the Federation or Higinbotham did a single thing that actually advanced nonproliferation, but they sure were smug about it…
I suspect that Al Copeland, by contrast, understood that he was a royal jerk. And he also understood that developing a chain of spicy chicken restaurants really does improve the human condition. Higinbotham’s failing was in mistaking self-righteous proclamations for actually making people’s lives better in a way that video games really do improve the human condition.
So, “The Higgy” will not identify the worst person in the world, just as “The Al” does not recognize the best. Instead, “The Higgy” will highlight individuals whose arrogant delusions of shaping the world to meet their own will outweigh the positive qualities they possess.
We will invite nominations for “The Higgy” in late March and will announce the winner, appropriately enough, on April 15. Thanks to Greg for his suggestions in developing “The Higgy.”