Jay P. Greene's Blog

There has been a flurry of research recently claiming to find compelling causal evidence that increasing school spending would significantly improve student outcomes and avoiding cuts in spending would prevent significant harm.  This research has been embraced so quickly as settled fact that over 400 researchers and advocates signed a group letter citing it while urging the federal government to provide financial support to local schools during the COVID recession. The confident conclusion that spending more is the path to improving education is so appealing that the research behind that claim has received remarkably little scrutiny.

A new study by Jessica Goldstein and Josh McGee begins to remedy this lack of skepticism by carefully attempting to replicate the most recent school finance study co-authored by Kirabo Jackson with Cora Wigger and Heyu Xiong, which is forthcoming in American Economic Journal: Economic Policy and has appeared in Education Next.  Jackson, Wigger, and Xiong examine the effect of K-12 spending cuts during the Great Recession by comparing the downturn in states where much of the funding comes from state revenue to states where more funding comes from local sources.  The idea is that state revenue is more sensitive to a recession, and so cuts would be more severe in states that were more reliant on state sources, even when the effects of the recession on the state’s economy were the same. Using this technique, they conclude that K-12 spending cuts hurt student outcomes.

Goldstein and McGee are able to reconstruct what Jackson, Wigger, and Xiong report, but they find that their results are highly sensitive to the non-standard ways in which they construct their statistical model and disappear or even change direction when trivial changes are made.  Goldstein and McGee also highlight some serious problems with the data used in the original study. 

Because these may sound like minor technical disputes, let me describe some of the issues in non-technical language so that readers can more easily grasp how much this replication effort undermines confidence in the original claims.  As Goldstein and McGee put it, “Econometric models can be constructed in a variety of ways, and many modeling choices may be somewhat arbitrary or theoretically unimportant. However, if the model’s estimates represent the true causal impact, they should be consistent across many different reasonable ways of constructing the model.”  The replication effort convincingly demonstrates that the original results claiming significant harms from spending cuts are not robust to these kinds of changes.  Of the many theoretically reasonable ways the original study could have constructed their model, its authors managed to find one that would yield significant positive results out of the many that would have yielded null results.

To compare states that are highly reliant on state revenue for K-12 spending to those that rely much less, Jackson, Wigger, and Xiong divide the 50 states and DC into three groups: those with more than 67% of K-12 spending coming from state sources, those with less than 33 percent coming from state sources, and all others in the middle.  Dividing states in this way places only four states in the high-reliance group and three in the low reliance group, with the remaining 44 states in the middle. The main results they present are based on the difference in outcomes between the top four and bottom three states.  This thin slice of states contains the two strange cases of DC and Hawaii, both of which only have a single school district and where state versus local revenue is not at all meaningful.  Goldstein and McGee try changing the thresholds for states being classified into the high and low categories to see if the results remain the same if they compare top versus bottom quartiles or deciles of states.  The exact grouping of states into high and low categories should not make much of a difference, but the replication shows that researchers would get null results if they had tried these reasonable alternative ways of categorizing states.

Similarly, the original study recognized that it is important to separate the effects of spending cuts in certain states from peculiar changes attributable to the time periods for all states.  Ideally, they would introduce a dummy variable for each year, which they say they tried but it yielded insignificant results.  Instead, they choose to group years into pre-recession, recession, and post-recession periods to control for idiosyncratic effects of changes over time.  The years that they label as pre, during, and post-recession, however, are not consistent with the official designation of the recession by the National Bureau of Economic Research.  So, the replication makes slight adjustments in how years are categorized and discover that doing so yields null results, sometimes with negative estimated effects of spending on student outcomes.  Again, real results should not disappear when these kinds of trivial changes are made.

The replication also considers the original study’s claim that spending cuts reduce college-going in the year following the spending change.  The theoretical mechanism by which this effect is produced is unclear given that college-going is likely the result of more than a decade of educational investment, not just the previous year’s spending.  Goldstein and McGee offer an alternative pathway by which college-going might be reduced, which is state expenditures on higher education.  As it turns out, states that rely heavily on state revenue for K-12 spending are also places where higher education relies heavily on state spending.  When those states cut K-12 spending during the Great Recession, they also cut higher education funding.  The replication substitutes higher education for K-12 spending in the original model, which yields similar effects on college-going rates.  This clearly demonstrates that the original study had not isolated the causal effect of K-12 spending cuts from the similar effects of higher education reductions. 

Lastly, the replication reveals several problems with the data used in the original study.  For example, the original study reports Vermont as having 68.3% of K-12 spending coming from state revenue while the Census, the data source they cite, indicates that figure should be 88.5%.  Similarly, Arkansas’ state share of spending is listed as 75.7%, which is consistent with the Census figure, but is almost 20 percentage points different from the number provided by the National Center for Education Statistics.  It is not obvious which is the better figure to use and these disparities reveal that identifying the state share of K-12 spending, on which the entire analysis depends, is problematic.  Most alarmingly, the results Jackson, Wigger, and Xiong produce in Figure 3 of their Education Next article claiming to show the effects of comparing results for states above and below the national median of reliance on state revenue could not be replicated by Goldstein and McGee (see Figures 7-9) and are almost certainly in error. Done correctly Figure 3 should show no effects on student outcomes from spending cuts.

It is disconcerting that neither the reviewers at AEJ-Policy or the over 400 researchers who signed the group letter were able to detect these data problems or raise questions about the unusual way in which states and years were grouped in the original study.  Before quickly embracing desired findings, the field needs to restore the traditional scientific virtues of skepticism and humility and apply those more generally to the new research on school spending effects.

The technocratic brand of ed reform that is currently dominant is based on the premise that policy elites, guided by science, need to ensure school quality.  Parents should have choices, but they should only choose among quality options.  Mostly using test scores, technocrats believe they can identify quality schools and quality-promoting educational practices, which should over-ride parental preferences about which schools and practices offer a quality education.

A new study by Diether W. Beuermann and C. Kirabo Jackson suggests that parents may be better at detecting which schools promote long-term positive outcomes for their children than technocrats guided by short-term test scores.  They examine the school system in the Barbados in which parents seek admission for their children into schools they prefer, but those schools use test-score cut-offs to determine which students gain admissions.  The cut-offs create a discontinuity that allows for a rigorous causal identification of whether students who barely gain admission to a desired school have different outcomes than those with barely lower lower test scores who are denied admission.

They find that test score gains are no greater for students who were admitted to the schools their parents preferred than those not admitted.  For boys there are some signs that the effect on test score gains may actually be negative.  But when they look at longer-term outcomes, including educational attainment, employment, and earnings, they find significant benefits for students who were admitted to the schools the parents preferred.  These positive effects were driven mostly by gains for girls.  When they explore mechanisms for why these gains occurred, they find a significant reduction in teen motherhood for girls admitted to preferred schools, which contributed to their educational attainment and later employment and earnings.  They also found that both boys and girls experienced significant long-term health benefits as measured by a healthy BMI, regular exercise, and dental check-ups if they gained admission to the schools their parents preferred.  The researchers conclude: “This suggests that preferred schools may promote productive habits and attitudes that are not measured by test scores but contribute to overall well-being. This may represent a significant, previously undocumented, return to school quality.”

So, parents, on average, could detect important aspects of school quality that technocrats guided by test scores would get wrong.  Technocrats would conclude that the schools that parents prefer do nothing to improve student outcomes because test scores don’t rise or even go down when students get into the school their parents want.  But parents are smarter than the technocrats.  They prefer schools that improve long-term outcomes for their children.  Specifically parents seem to be able to choose schools that are more effective in developing the “character” of their children, making the students less likely to get pregnant as teens and more likely to be engaged in positive health behaviors later.  For boys this may not make a big difference in the labor market (although it does not harm those outcomes), but for girls these health improvements seem to drive higher educational attainment, employment, and earnings.

This study is consistent with a long line of research that finds a disconnect between short-term test score outcomes and long-term life outcomes, as described in a recent meta-analysis by my colleagues, Mike McShane, Pat Wolf, and Collin Hitt.  It’s amazing to me how champions of the technocratic approach continue to have faith that they have access to scientific tools to identify school quality that less well-informed parents lack despite the growing body of scientific evidence that demonstrates the very real defects of the technocratic approach.  Despite their daily hymns of praise to science, the technocrats don’t seem very scientific at all.

(Guest Post by Dr. Williamson M. Evers)

Improving education is a combination of (1) teachers effectively conveying to students certain essential information; (2) getting a better match between schools (with different strengths and teaching talents) and students (with different capacities, needs, and interests); and (3) creating learning environments where teachers are motivated to engage the students’ intellects and emotions and students are motivated to learn challenging subject-matter — and learning in a way that embeds it and puts it into practice beyond the level of learning facts that could be looked up on the web. The classroom should be a kind of theater of high-morale learning. In that shared theater of learning, the important components of civilization are passed on, teachers are role models for students, and students’ character is formed.

Curriculum should promote patriotic and liberty-loving citizenship (without ruling out exposure in high school to our country’s problems.) We should always remember that the family is the site and source of the most important education that a child receives. The public school system should avoid undermining the family and related social institutions like churches, charities, and voluntary associations.

There are steps that Alabama should consider to improve student performance:

Reading improvement. The Alabama Reading Initiative (ARI) was a professional development program (started in 1998) with school-level reading coaches. It took a few years to plan and put into place. It was based on the scientific research literature on the teaching of reading and therefore took a phonics-first approach.

By 2007, Alabama’s grade 4 reading score on the National Assessment of Educational Progress (NAEP), a nationwide examination that tests a sample of students, had risen significantly – an eight-point jump from 2005 (almost a grade level). The grade 4 students made another four-point jump in 2011, catching up with the national average. In 2009, Alabama’s grade 8 reading score on NAEP rose three points, and in 2011, it rose another three points. The grade 8 gains were slower and less sharp than the gains for grade 4 and remained  five to six points below the national average. Even the less dramatic grade 8 gains did constitute considerable improvement.

The official history of the ARI says:

“From 2003 to 2011, with a state-funded reading coach in every elementary school, Alabama’s 4th graders made more progress in reading on the National Assessment of Educational Progress (NAEP) than students from any other state in the nation. Alabama met the national average in 4th grade reading for the first time, and Alabama is one of only four states in the nation (second only to Maryland) to show increases in 4th grade reading from 2009 to 2011. The number of Alabama students reading below grade level has been reduced by half.”

Since the coming of Common Core, the State Superintendent has directed the funds for the reading initiative and the STEM initiative to coaches for Common Core. Subsequently, grade 4 reading scores on NAEP in Alabama have declined eight points, to below the national average. Grade 8 reading scores remain stagnant.

I don’t like to made pronouncements about policy issues without talking to a range of people and looking into the details of what changes could be made. But my working hypothesis would be that the reading initiative needs to be restored.

STEM (science, technology, engineering, and mathematics) improvement. Alabama also had a long-running math, science, and technology initiative. In contrast with the reading initiative, it seems to have produced no noticeable effects on student achievement. I think it makes sense for Alabama to look into why there has been such a sharp difference in results between the two initiatives, and make appropriate adjustments.

In both STEM and reading, the Alabama State Department of Education should consider certifying Professional Development programs that are based on science – based on empirical research with control groups and the like. A State superintendent should seek to make sure that districts know about good Professional Development and should nurture the growth of such programs.

Graduation rate and academic attainment. To boost academic attainment, Alabama should put in place a 3^rd grade test (Florida currently retains students who cannot pass its 3^rd grade reading test) and a 10^th-grade/graduation test. Alabama discarded its High School Exit Examination (AHSGE – Alabama High School Graduation Exam) in 2013 and replaced it with ACT/CCRS (College- & Career-Ready Standards).

The 11^th grade ACT cut scores in Alabama are 18 for English; 22 for math and reading; and 23 for science. The ACT scores are the standard current national “college readiness” scores. So they are, in fact, quite demanding and above the 10^th grade level.

ACT is basically used in Alabama for federal reporting purposes to comply with the federal Elementary & Secondary Education Act.  Graduation is currently based on area requirements and seat-time (measured in Carnegie units).

Alabama should restore its exit exam, but it should operate in a multiple second-chances way, with many opportunities to retake the tests and offering a variety of exemptions and special categories.  The idea is to create a focus and a shared goal, not to deny some large number of students a diploma.  Alabama should also reconsider its One Diploma policy.  Since there are a wide variety of students with different capacities, interests, and needs, there should probably be a variety of diplomas.

Tom Loveless of the Brookings Institution estimates that all the prep time and all the millions spent on the Common Core national curriculum-content standards boosted achievement in the U.S. by only a tiny amount: one scale-score point. The new Every Student Succeeds Act does much to protect against future federal interference with curriculum. When it comes to testing, Alabama is not now with PARCC or SmarterBalanced, but has instead affiliated with ACT. Yet Alabama has a set of state standards largely based on Common Core. It is time, I would think, to revisit those current state standards to see what needs fixing. (I participated in California’s line-by-line review of the Common Core in 2010, and I participated in creating the late-1990s California State Academic Content Standards, judged best in the nation by the Thomas B. Fordham Foundation and the American Federation of Teachers.)

Career and technical education. Students, because they are different individuals, should have different educational pathways in high school. Blouke Carus — a leading children’s magazine publisher (including Cricket, Ladybug, Cobblestone), math and reading textbook developer, and chairman emeritus of the Carus Corporation (a company producing manganese compounds) — has said: “Our schools need to offer each student a choice among six or more challenging and rigorous high school curricula, as do other, higher-performing countries.”

As I and my colleagues wrote in a 2013 manifesto entitled “Closing the Door on Innovation”: “There is no evidence to justify a single high school curriculum for all students. A single set of curriculum guidelines, models, or frameworks cannot be justified at the high school level, given the diversity of interests, talents, and pedagogical needs among adolescents. American schools should not be constrained in the diversity of the curricula they offer to students. Other countries offer adolescents a choice of curricula; Finland, for example, offers all students leaving grade 9 the option of attending a three-year general studies high school or a three-year vocational high school, with about 50% of each age cohort enrolling in each type of high school. We worry that the ‘comprehensive’ American high school may have outlived its usefulness, as a [2011] Harvard report [Pathways to Prosperity] implies.”

Because vocational education has a reputation for being a dumping ground and a place where minority children were channeled because of racial bias, education policy has neglected vocational education and sometimes even eliminated it. Alabama’s “Essentials/Life Skills Pathway” is a separate vocational-education pathway.  For example, instead of English 9, the student would take English Essentials 9. This is intended to prep students for community college or the workforce, not for entry to 4-year colleges.

Alabama needs to avoid a situation which the only high school available for a student is a college-prep comprehensive or, alternatively, vocational-only.  The state should incentivize local districts (through waivers) to create a variety of career-tech program alternatives (including a variety of mission-oriented schools), but these should include an academic component (as they do now). In general, Alabama should look carefully at Arizona where the process of school creation is highly decentralized, and the results have been fruitful.

Teacher Quality. As the 1999 manifesto “The Teachers We Need” (of which I was a co-signer) says, “the surest route to quality is to widen the entryway [and] deregulate the processes.” This would expand the pool of potential teachers. It would open the doors to highly-qualified, but non-traditional recruits who want to become teachers and thereby improve overall quality. New York State currently has a relatively open process allowing alternative certification.  In Arkansas, almost half the incoming teachers come in under alternative certification.

But Alabama gets only a C- from the National Council on Teacher Quality for the extent to which the state’s teaching pool has been expanded by alternative certification and the like. Alabama should also be open to non-traditional teacher training programs like the Relay Graduation School of Education in New York City.

Michael McShane of the American Enterprise Institute has written:

“Recently, I have been influenced by the work of Northwestern University economist Kirabo Jackson, whose fascinating NBER working paper calls into serious question policy’s recent overreliance on math and reading scores as the primary measure of the ‘goodness’ of schools and teachers. As it turns out, teachers have important and measurable impacts on both the cognitive and non-cognitive development of students. While it’s certainly true that test scores can tell us something important about a teacher, what is troubling for the test-score types is that it looks like (1) non-cognitive scores are better predictors of later life success (completing high school, taking the SAT, and going to college) and (2) that it is not the same set of teachers that is good at raising both cognitive and non-cognitive measures.

“Such has to be the same for schools, right? If there are teachers that are increasing non-cognitive, but not cognitive skills, surely there are schools that are doing the same. As a result, trying to assess if a school is ‘good’ or ‘bad’ relies on a complex web of preferences and objective measures that, quite frankly, cannot be taken into account in a centralized accountability system. We need something more sophisticated, and something that can respect a diverse conception of what ‘good’ and ‘bad’ means.”

In citing Michael McShane and Kirabo Jackson, I am suggesting a goal for the future, while not saying that there should be no accountability. We should consider a change in providers of accountability because Alabama has a variety of school districts, localities, and other groupings. As Jason Bedrick of the Cato Institute and Andrew Smarick, chair of the Maryland State Board of Education, have separately pointed out, accountability doesn’t have to be in conformity with One Best Way. School evaluation can be pluralistic and rivalrous and doesn’t have to be solely governmental. Churches could do it, neighborhood associations could do it, chambers of commerce could do it.   As Bedrick writes: “Parents can then evaluate the quality of education providers based on their own experience and the expert evaluations of appropriate external providers, and the entire system evolves as parents select the providers that best meet their children’s needs.”

Currently if a teacher is educated in Alabama teachers colleges and receives bad evaluations during the first three years, the teacher can go back to college for free. This seems like a smart program worth retaining.

Education Next magazine surveyed teachers nationwide and the teachers themselves said that about 5% of teachers deserve an “F” and 8% deserve a “D.” But we should be cautious about grandiose projects to improve teacher quality. The “highly qualified teacher” provisions of the No Child Left Behind law were never successfully put into effect. The elaborate new teacher-evaluation systems put in place (at the cost of millions) in about 20 states have reduced the percentage of teachers rated “satisfactory” from 99% to 97%.

Currently teacher-certification testing is under court supervision. Alabama has low passing scores on the PRAXIS II tests that it expects of teachers. When court supervision is over, the state could slowly raise passing scores (as Texas has). Teachers do need to know the subject-matter they are teaching. But the most important thing is to increase the inflow of good prospects into the teaching pool (from which districts and schools can select), rather than reducing the size of the pool.

Economic growth and student attainment. Both policymakers and the civically active public underestimate the economic gains from school improvement. The differences in rates of growth among states can be matched to the education of the workers in the various states. A multiplicity of causes affect economic growth, but nothing is more influential in the long run than school improvement. Although it is certainly difficult to improve schools, it is easier to improve schools than it is to change other factors that figure in increased productivity.

Equated to PISA international test results, Alabama is currently below Turkey.

Some people think that Alabama’s low results are because of demographics. But the only state whose whites do worse currently is West Virginia.

For Alabama, which is educationally at the bottom of the lowest quartile in the United States, the present value of bringing the state to the level of Kentucky – in the next quartile — would be three times Alabama’s current Gross Domestic Product (according to my Hoover Institution colleague Eric A. Hanushek). Other states at the Kentucky level are Florida, Arizona, Oklahoma, and Rhode Island. To get to the level of Kentucky would mean for Alabama over 6% annual growth in GDP and about 12% annual growth in salaries.

Conclusion: A State Superintendent should not be a Caesar, lording it over local districts and local superintendents. State superintendents should not act as if they are progressive technocrats with coercive utopian powers. It will be harmful to learning communities if state superintendents act that way.  Where sensible, they should devolve much of the responsibility over academic content, teaching methods, and instructional materials to the local districts. The different districts can – and should — try different things. Parallel learning communities will sometimes arise.  If districts and schools endeavor to conscientiously do their best, Alabama can climb out of the cellar and surpass states like Kentucky academically. The State of Alabama will prosper, and Alabama’s children will have more fulfilling lives.

—————————————————————

Williamson M. Evers, Ph.D., is a Research Fellow at Stanford University’s Hoover Institution and a former U.S. Assistant Secretary of Education for Planning, Evaluation, and Policy Development. He is a finalist for the position of Alabama State Superintendent of Education.

(Guest Post by Jason Bedrick)

Every now and then, Thomas Sowell writes a column titled “Random Thoughts on the Passing Scene” where he offers up gems like this:

Stupid people can cause problems, but it usually takes brilliant people to create a real catastrophe.

I’m no Thomas Sowell, but here are a few of my own (much less pithy or clever) random thoughts the passing education policy scene:

Montana Department of Revenue: Religious Families Need Not Apply

Last October, the Montana Department of Revenue interpreted the state’s constitution to prohibit it from issuing tax-credit scholarships to students attending religious schools. Fortunately, a judge ordered them to reverse course after the heroes at the Institute for Justice — which deserves its title as the “nation’s pre-eminent courtroom defender of school choice” — filed suit. The injunction is only temporary, pending the outcome of the case, but the MT-DOR bureaucrats just filed an appeal. Can’t let parents get a taste of choice! Why, they might choose something else!

Sadly, they’re far from the only ones working to block educational choice…

The Left v. Educational Choice

Lawsuits against two of the most ambitious educational choice policies are each one step closer to resolution. In Florida, the teachers’ union (joined by the NAACP and others) is challenging the state’s tax-credit scholarship program, which served nearly 70,000 low-income students. A lower court tossed out the challenge based on standing but the union appealed. Earlier this week, an appeals court heard oral arguments in the case, where the union’s attorney struggled to explain how the choice law harms anyone:

Lynn Hearn, arguing for the groups challenging the program, said students receiving scholarships are spread unevenly across the state. The program affects public schools “by drawing students out of the system and sending the funding away,” she said, and schools don’t always reap savings from having fewer children to educate.

“You’ll have a few from this school and a few from that school, and so the school is left with exactly the same expenses,” she said.

Judge Lori Rowe, part of a three-judge panel that heard the case, was skeptical.

“You haven’t alleged that any individual student is suffering,” she said to Hearn. “You haven’t alleged that per-student funding has been reduced. You haven’t even alleged that the [state] education budget has been reduced.”

Meanwhile, the Nevada Supreme Court is preparing to hear oral arguments in a lawsuit challenging the state’s education savings account law. Last month, the attorney general’s office “formally nudged” a judge to rule on a separate challenge by the ACLU that was filed back in August. There’s a decent chance that the state supreme court will merge the two cases.

Why is the left so determined to keep families from exercising choice? In short, it’s about control. And that leads me to…

The Bathroom Wars

It’s frankly incredible that with all the serious problems this country is facing, the president thinks it’s a priority to issue a decree forcing public schools to let children pick which bathroom–or locker room–they want to use. Yes, that’s right, the feds have ordered public schools to give biological males access to the girls’ locker room, where teenage girls undress and shower. The White House press secretary, the Orwellianly named Josh Earnest, claims that the new decree is only “guidance” not a “threat,” but if schools don’t follow the “guidance” then the Obama administration promises to cut their federal funding.

Late last year, the feds intervened when a public school in Illinois required a transgender student who is biologically male to “change and shower separately from her teammates and classmates.” Rather than force the student to use the boys’ locker room, the school came to a compromise that attempted to respect the privacy of all the students involved. Nevertheless, the feds filed suit, demanding the transgender student receive “unfettered access” to the girls’ locker room, even though some teenage girls expressed discomfort undressing and showering with someone who is anatomically male. Eventually, the school caved. Outraged parents have filed a lawsuit seeking to overturn the school’s agreement with the feds, and this week the ACLU announced it was intervening to support the feds.

As Neal McCluskey (my wise and benevolent boss at the Cato Institute) often observes, the zero-sum nature of political control over schooling forces citizens into social conflicts like the Bathroom Wars. A much less divisive alternative would be (you guessed it!) a system of educational choice in which parents could select the schools that have their preferred bathroom policy:

As important, if not more so, is that allowing private entities to choose their own policies is consistent with individual liberty, including freedom of association and religion, while it is much better suited to enabling people with competing values to peacefully co-exist. There is no zero-sum contest: Those who want an open bathroom policy could choose schools in which all the staff and families also embraced it, while those feeling more comfortable with bathrooms and locker rooms restricted by biological sex could go to schools with like-minded people.

Perhaps the best examples of educational choice helping to bring peace and balance rights have been in many European countries, where religious conflicts in schools abated as governments decided to fund choices of Protestant, Catholic, nonsectarian, or other institutions. Getting to the place of greater peace requires something difficult – accepting that all people should be able to live as they want as long as they do not force themselves on others, and even if we do not like the choices they make – but living and letting live is the foundation of a free society.

But forget bathrooms–schools might want to consider separating the genders in the classroom as well…

New Study: Single-Sex Schooling Works

A new study by Prof. C. Kirabo Jackson of Northwestern University finds that single-sex schooling improves student outcomes:

The results show that single-sex education can improve both boys’ and girls’ outcomes. Three years after being assigned to a single-sex secondary school, both boys and girls have higher scores on standardized tests. Five years later, they are more likely to take and pass advanced courses. In the long run, both boys and girls are more likely to have completed secondary school and to have earned the credential required to continue to tertiary education. Importantly, boys are also less likely to have been arrested. Taken as a whole, the results suggest that being in the single-sex cohorts improved test scores and also improved longer-run non test score outcomes such as advanced course taking, high school completion and engaging in criminal activity.

As AEI’s Michael Strain highlighted, “the benefits of single-sex instruction are free to the taxpayer — all you have to do is sort children into the appropriate classrooms or schools.”

You won’t be surprised to learn that, evidence be damned, the ACLU opposes this as well.

Here’s my first round in the debate with Mike Petrilli over whether test scores are reliable indicators of quality that can be used by regulators and policymakers to identify schools to be closed or expanded…

——————————————————-

The school choice tent is much bigger than it used to be. Politicians and policy wonks across the ideological spectrum have embraced the principle that parents should get to choose their children’s schools and local districts should not have a monopoly on school supply.

But within this big tent there are big arguments about the best way to promote school quality. Some want all schools to take the same tough tests and all low-performing schools (those that fail to show individual student growth over time) to be shut down (or, in a voucher system, to be kicked out of the program). Others want to let the market work to promote quality and resist policies that amount to second-guessing parents.

In the following debate, Jay Greene of the University of Arkansas’s Department of Education Reform and Mike Petrilli of the Thomas B. Fordham Institute explore areas of agreement and disagreement around this issue of school choice and school quality. In particular, they address the question: Are math and reading test results strong enough indicators of school quality that regulators can rely on them to determine which schools should be closed and which should be expanded—even if parental demand is inconsistent with test results?

To a very large degree, education reform initiatives hinge on the belief that short term changes in reading and math achievement test results are strong predictors of long term success for students. We use reading and math test scores to judge the quality of teachers, schools, and the full array of pedagogical, curricular, and policy interventions. Math and reading test scores are the yardstick by which education reform is measured. But how good of a yardstick is it?

Despite the centrality of test scores, there is surprisingly little rigorous research linking them to the long-term outcomes we actually care about. The study by researchers from Harvard and Columbia (Chetty, et al.) showing that teachers who increase test scores improve the later-life earnings of their students is a notable exception to the dearth of evidence on this key assumption of most reform initiatives. But that is one study, it has received some methodological criticism (although I think that has been addressed to most people’s satisfaction), and its results from low-stakes testing may not apply to the high-stakes purposes for which we would now like to use them. This seems like a very thin reed on which to rest the entire education reform movement.

In addition, we have a growing body of rigorous research showing a disconnect between improving test scores and improving later-life outcomes. I’ve written about this at greater length elsewhere (see here and here), but we have eight rigorous studies of school choice programs in which the long-term outcomes of those policies do not align with their short-term achievement test results. In four studies, charter school programs that produce impressive test score gains appear to yield no or little improvement in educational attainment. In three studies of private school choice and one charter school choice program, we observe large benefits in educational attainment and even earnings but little or no gains in short-term test score measures.

If policy analysts and the portfolio managers, regulators, and other policy makers they advise were to rely primarily on test scores when deciding which programs or schools to shutter and which to expand, they would make some horrible mistakes. Even if we ignore the fact that most portfolio managers, regulators, and other policy makers rely on the level of test scores (rather than gains) to gauge quality, math and reading achievement results are not particularly reliable indicators of whether teachers, schools, and programs are improving later-life outcomes for students.

What explains this disconnect between math and reading test score gains and later-life outcomes? First, achievement tests are only designed to capture a portion of what our education system hopes to accomplish. In particular, they are not designed to measure character or non-cognitive skills. A growing body of research is demonstrating that character skills like conscientiousness, perseverance, and grit are important predictors of later-life success (see this, for example). And more recent research by Matt Kraft, Kirabo Jackson, and Albert Cheng and Gema Zamarro (among others) shows that teachers, schools, and programs that increase character skills are not necessarily the same as those that increase achievement test results. There are important dimensions of teacher, school, and program quality that are not captured by achievement test results. Second, math and reading achievement tests are not designed to capture what we expect students to learn in other subjects, such as science, history, and art. Prioritizing math and reading at the expense of other subjects that may be important for students’ later-life success would undermine the predictive power of those math and reading results. Third, many schools are developing strategies for goosing math and reading test scores in ways that may not contribute to (and may even undermine) later-life success. The fact that math and reading achievement results are overly narrow and easily distorted makes them particularly poor indicators of quality and weak predictors of later-life outcomes.

I do not mean to suggest that math and reading test results provide us with no information or that we should do away with them. I’m simply arguing that these tests are much less reliable indicators of quality than most policy analysts, regulators, and policy makers imagine. We should be considerably more humble about claiming to know which teachers, schools, and programs are good or bad based on an examination of their test scores. If parents think that certain teachers, schools, and programs are good because there is a waiting list demanding them, we should be very cautious about declaring that they are mistaken based on an examination of test scores. Even poorly educated parents may have much more information about quality than analysts and regulators sitting in their offices looking at spreadsheets of test scores.

I also do not mean to suggest that policy makers should never close a school or shutter a program in the face of parental demand. I’m just arguing that it should take a lot more than “bad” test scores to do that. Yes, parents can and will make mistakes. But analysts, authorizers, regulators, and other policy makers also make mistakes, especially if they rely predominantly on test results that are, at best, weak predictors of later-life success. The bar should be high before we are convinced that the parents are mistaken rather than the regulators poorly guided by test scores. Besides, we should prefer letting parents make mistakes for their own children over distant bureaucrats making mistakes for hundreds or thousands of children while claiming to protect them.

(Also posted at Flypaper )

(Guest Post by Eric A. Hanushek)

Considerable prior research has failed to find a consistent relationship between school spending and student performance, making skepticism about such a relationship the conventional wisdom. Given that skepticism, new studies that purport to find a systematic relationship between school spending and student performance get disproportionate attention.

There is in fact great demand for results linking funding with favorable outcomes.  Knowing that a strong relationship existed would mean that policy makers outside of the schools – legislatures, governors, and courts – only have to concern themselves with how much money was provided to schools and not with how money was used.  And, meeting our education challenges by providing more money appears from history to be easier than pursuing more fundamental changes in schools.

Kirabo Jackson, Rucker C. Johnson, and Claudia Persico offer a new study suggesting that a clear money-performance relationship exists if you just look in the right place. Their overarching conclusion is that “methods matter.” Their discovery of a money-performance relationship is attributed to analyzing the effects of spending that emanates from court decisions (exogenous variation in spending), tracing the effect of this spending to long run outcomes (completed schooling and wages), and focusing on the right subgroup (disadvantaged students).

From a methodological viewpoint, details are important here. How court decisions are dated given long and repeated legal involvement in many states; how the spending reaction to court decisions is measured; whether the court decisions are unrelated to the character of schools before court involvement; and how court-mandated spending differs from other increased spending are a few of the details.  Nevertheless, while these are important methodological issues, it is more useful to focus on the substance of their findings.

Jackson, Johnson, and Persico reach the following conclusions about the impact of a 10 percent increase in spending for all 12 years of schooling:  1) It would increase years of schooling by 0.44 years for poor children and by an insignificant 0.075 years for non-poor children, implying that a spending increase of 22.7 percent would eliminate the average education gap; 2) It would increase high school graduation rates by 11.6 percentage points for poor children and 6 percentage points for non-poor children; 3) It would increase subsequent family incomes by 16.4 percent for poor children and zero for non-poor children; 4) It would reduce subsequent adult poverty rates by 6.8 percentage points for poor children and zero for non-poor children.

Their analysis covers schooling experiences for the period 1970-2010.  Thus, it is useful to connect these estimates to actual funding patterns over the period.  Between 1970 and 1990, real expenditure per pupil increased not by 10 percent but by over 84 percent.  By 2000, this comparison with 1970 topped 100 percent, and it reached almost 150 percent by 2010.  No amount of adjustment for special education, LEP, or what have you will make these extraordinary increases in school funding go away.

If a ten percent increase yields the results calculated by Jackson, Johnson, and Persico, shouldn’t we have found all gaps gone (and even reversed) by now due to the actual funding increases?  And, even with small effects on the non-poor, shouldn’t we have seen fairly dramatic improvements in overall educational and labor market outcomes? In reality, in the face of dramatic past increases in school funding, the gaps in attainment, high school graduation, and family poverty have remained significant, largely resisting any major improvement.  And, the stagnating labor market performance for broad swaths of the population has captured considerable recent public and scholarly attention.

What could reconcile these apparent inconsistencies?  Here are some possibilities:

• There might be sharply diminishing returns to spending so that their estimates apply most clearly in 1970 when spending (in 2011-12 $’s) was just $4,500 as opposed to 2010 when spending was $11,000. Thus, by implication, spending today might be expected to have a much smaller impact than they estimate.

• Only spending induced by the courts might have the large impacts they identify, with spending not related to judicial rulings having a negligible impact. The likelihood of this is a little questionable since, with a few exceptions such as NJ and WY, the courts have done little to look into how any legislature responds in terms of specific spending programs.  But, if true, normal spending increases by state legislatures and by local taxpayers would not be expected to have any impact on outcomes.

• The estimates of Jackson, Johnson, and Persico might simply be very wrong.

Maybe there are other ways to reconcile the Jackson, Johnson, and Persico estimates with the aggregate data on spending and outcomes.  But in the end they themselves state what is now commonly accepted: How money is spent matters.  Indeed, by simple consideration of their evidence, how money is spent is more important than how much is spent.

Of course, it is always important to recognize that none of this discussion suggests that money never matters.  Or that money cannot matter.  It just says that the outcomes observed over the past half century – no matter how massaged – do not suggest that just throwing money at schools is likely to be a policy that solves the significant U.S. schooling problems seen in the levels and distribution of outcomes.  We really cannot get around the necessity of focusing on how money is spent on schools.

This is the question raised by a new study by C. Kirabo Jackson, Rucker C. Johnson and Claudia Persico  in Education Next.  Jackson, et al claim to have up-ended decades of school finance research by finding a link between school spending and improved student outcomes.  After reading that article and an earlier, more detailed version posted on the NBER web site, I find nothing to persuade me to abandon the long-standing and well-established finding that simply providing schools with more resources does not improve student outcomes.

Let’s remember how well-established this finding is by noting that Eric Hanushek conducted a comprehensive review of the literature and concluded:

…the research indicates little consistent relationship between resources to schools and student achievement. Much of the research considers how resources affect student achievement as measured by standardized test scores. These scores are strongly related to individual incomes and to national economic performance, making them a good proxy for longer run economic impacts. But, the evidence – whether from aggregate school outcomes, econometric investigations, or a variety of experimental or quasiexperimental approaches – suggests that pure resource policies that do not change incentives are unlikely to be effective. (p. 866)

Jackson, et al acknowledge that past research has failed to find a link between school resources and student outcomes:

Coleman found that variation in school resources (as measured by per-pupil spending and student-to-teacher ratios) was unrelated to variation in student achievement on standardized tests. In the decades following the release of the Coleman Report, the effect of school spending on student academic performance was studied extensively, and Coleman’s conclusion was widely upheld.

But they believe that past research was flawed in two important respects.  First, test scores may be a weak indicator of later-life success, so it would be better to look at stronger measures, like educational attainment, employment, and earnings.  Second, they believe that past studies of school spending may suffer from an endogeneity problem.  That is, extra money has tended to go to schools facing challenges.  The failure to find a link between more resources and better achievement may be because schools with a weaker future trajectory are the ones more likely to get more money.  So, the causal arrow may be going in the wrong direction.  Weak performance may be causing more resources rather than more resources causing weak performance.

Jackson, et al solve the first issue by focusing on longer-term student outcomes, like educational attainment and earnings.  They claim to have a solution to the second problem by finding a type of spending increase that is unrelated to the expected trajectory of school performance.  Court-ordered spending, they say, is exogenous, while regular legislative increases in spending are endogenous.

The surprising findings of the Jackson, et al article hinge entirely on this claim that court-ordered spending is exogenous.  Looking at attainment and earnings by itself does not produce a different result than past research that has focused on test scores.  The thing that allows Jackson, et al to find that spending is linked to better student outcomes is the fact that they do not examine actual spending increases.  Instead, they predict changes in spending based on court-orders and use that predicted spending in place of the actual spending.

This instrumental variable technique developed by James Heckman, however, only works if the instrument is in fact exogenous.  That is, court-ordered spending has to be unrelated to the future trajectory of school performance.  Given how critical this point is to the entire article, you might think Jackson, et al would spend a fair amount of energy to justifying the exogeneity of court-ordered spending.  They do not.

It is completely mysterious to me why we should believe that court-ordered spending differs from legislatively-originated spending in the likelihood that it is linked to the expected future trajectory of school performance.  That is, schools facing challenges are just as likely to get extra money if the spending originates in the courts or in the legislature.  If we are concerned that the causal arrow is going in the wrong direction in that weak performance causes more money rather than the other way around, we should have that concern just as much whether the motivation for the money came from the court or the legislature.

Jackson, et al do not make a proper case for the exogeneity of court-ordered spending other than to describe it as a “shock” to school spending.  But there is nothing more shocking about spending that originates in the courts than in the legislature.  Court cases take years to develop, be decided, and complete appeals.  And then they have to be implemented by legislative action.  The timing of court-ordered spending is no more surprising to schools than regular legislative spending.  Nor is the amount of spending change necessarily more dramatic than those originating in legislatures.  The passage of ESEA and its re-authorizations infused large amounts of money into schools.

Jackson et al need to convince us that court-ordered spending is exogenous to get their unusual result.  If they just used conventional methods, they would confirm the wide-spread finding that extra money does not improve outcomes.  As they describe it:

We confirm that our approach generates significantly different results than those that use observed increases in school spending, by comparing our results to those we would have obtained had we used actual rather than predicted increases as our measure of changes in district spending. For all outcomes, the results based simply on observed increases in school spending are orders of magnitude smaller than our estimates based on predicted SFR-induced spending increases, and most are statistically insignificant.

But Jackson, et al fail to justify the claim that court-ordered spending is exogneous on which their entire article depends nor does such a claim seem plausible.

But even if you were to somehow believe that court-ordered spending is exogenous, it would still be unwise to jump to the conclusion that we now know money matters and should open the resource spigots to K-12 education.  First, the past research Hanushek reviewed includes studies that do not suffer from either of the concerns raised by Jackson, et al.  That is, some of those studies examine later-life outcomes for students and not just test scores and some of those studies rely on experimental methods with which there is no problem with causation.  Why should we disregard those studies for this one new study even if we were to ignore the concerns I’ve raised above?

Second, Jackson, et al are examining the effect of court-ordered spending in the 1970s when spending levels in real terms were much lower and variation in spending across districts within states was much higher.  It’s quite a leap to think that more money now would have the same effect as then.  To my surprise, Bruce Baker made this same point in response to the Jackson, et al article in comments to Education Week:

“[E]xploring such [far-apart] outcomes, while a fun academic exercise, is of limited use for informing policy,” he wrote in an email to Education Week. “Among other things, these are changes that occurred under very different conditions than today.”

Mr. Baker also disagreed with the researchers’ caveat that similar changes might have a much smaller effect if introduced today, in part because total school funding nationwide increased by 175 percent over 43 years, from an average of $4,612 per student in 1967 to about $12,772 per student in 2010, as measured in 2012 dollars.

So does school spending matter after all?  I think the answer is still clearly “no.”