Parents are Smart. Technocrats are Dumb

July 12, 2018

Image result for jerry lewis professor

The technocratic brand of ed reform that is currently dominant is based on the premise that policy elites, guided by science, need to ensure school quality.  Parents should have choices, but they should only choose among quality options.  Mostly using test scores, technocrats believe they can identify quality schools and quality-promoting educational practices, which should over-ride parental preferences about which schools and practices offer a quality education.

A new study by Diether W. Beuermann and C. Kirabo Jackson suggests that parents may be better at detecting which schools promote long-term positive outcomes for their children than technocrats guided by short-term test scores.  They examine the school system in the Barbados in which parents seek admission for their children into schools they prefer, but those schools use test-score cut-offs to determine which students gain admissions.  The cut-offs create a discontinuity that allows for a rigorous causal identification of whether students who barely gain admission to a desired school have different outcomes than those with barely lower lower test scores who are denied admission.

They find that test score gains are no greater for students who were admitted to the schools their parents preferred than those not admitted.  For boys there are some signs that the effect on test score gains may actually be negative.  But when they look at longer-term outcomes, including educational attainment, employment, and earnings, they find significant benefits for students who were admitted to the schools the parents preferred.  These positive effects were driven mostly by gains for girls.  When they explore mechanisms for why these gains occurred, they find a significant reduction in teen motherhood for girls admitted to preferred schools, which contributed to their educational attainment and later employment and earnings.  They also found that both boys and girls experienced significant long-term health benefits as measured by a healthy BMI, regular exercise, and dental check-ups if they gained admission to the schools their parents preferred.  The researchers conclude: “This suggests that preferred schools may promote productive habits and attitudes that are not measured by test scores but contribute to overall well-being. This may represent a significant, previously undocumented, return to school quality.”

So, parents, on average, could detect important aspects of school quality that technocrats guided by test scores would get wrong.  Technocrats would conclude that the schools that parents prefer do nothing to improve student outcomes because test scores don’t rise or even go down when students get into the school their parents want.  But parents are smarter than the technocrats.  They prefer schools that improve long-term outcomes for their children.  Specifically parents seem to be able to choose schools that are more effective in developing the “character” of their children, making the students less likely to get pregnant as teens and more likely to be engaged in positive health behaviors later.  For boys this may not make a big difference in the labor market (although it does not harm those outcomes), but for girls these health improvements seem to drive higher educational attainment, employment, and earnings.

This study is consistent with a long line of research that finds a disconnect between short-term test score outcomes and long-term life outcomes, as described in a recent meta-analysis by my colleagues, Mike McShane, Pat Wolf, and Collin Hitt.  It’s amazing to me how champions of the technocratic approach continue to have faith that they have access to scientific tools to identify school quality that less well-informed parents lack despite the growing body of scientific evidence that demonstrates the very real defects of the technocratic approach.  Despite their daily hymns of praise to science, the technocrats don’t seem very scientific at all.

 


How to Improve Alabama’s Schools

August 8, 2016

(Guest Post by Dr. Williamson M. Evers)

Improving education is a combination of (1) teachers effectively conveying to students certain essential information; (2) getting a better match between schools (with different strengths and teaching talents) and students (with different capacities, needs, and interests); and (3) creating learning environments where teachers are motivated to engage the students’ intellects and emotions and students are motivated to learn challenging subject-matter — and learning in a way that embeds it and puts it into practice beyond the level of learning facts that could be looked up on the web. The classroom should be a kind of theater of high-morale learning. In that shared theater of learning, the important components of civilization are passed on, teachers are role models for students, and students’ character is formed.

Curriculum should promote patriotic and liberty-loving citizenship (without ruling out exposure in high school to our country’s problems.) We should always remember that the family is the site and source of the most important education that a child receives. The public school system should avoid undermining the family and related social institutions like churches, charities, and voluntary associations.

There are steps that Alabama should consider to improve student performance:

Reading improvement. The Alabama Reading Initiative (ARI) was a professional development program (started in 1998) with school-level reading coaches. It took a few years to plan and put into place. It was based on the scientific research literature on the teaching of reading and therefore took a phonics-first approach.

By 2007, Alabama’s grade 4 reading score on the National Assessment of Educational Progress (NAEP), a nationwide examination that tests a sample of students, had risen significantly – an eight-point jump from 2005 (almost a grade level). The grade 4 students made another four-point jump in 2011, catching up with the national average. In 2009, Alabama’s grade 8 reading score on NAEP rose three points, and in 2011, it rose another three points. The grade 8 gains were slower and less sharp than the gains for grade 4 and remained  five to six points below the national average. Even the less dramatic grade 8 gains did constitute considerable improvement.

The official history of the ARI says:

“From 2003 to 2011, with a state-funded reading coach in every elementary school, Alabama’s 4th graders made more progress in reading on the National Assessment of Educational Progress (NAEP) than students from any other state in the nation. Alabama met the national average in 4th grade reading for the first time, and Alabama is one of only four states in the nation (second only to Maryland) to show increases in 4th grade reading from 2009 to 2011. The number of Alabama students reading below grade level has been reduced by half.”

Since the coming of Common Core, the State Superintendent has directed the funds for the reading initiative and the STEM initiative to coaches for Common Core. Subsequently, grade 4 reading scores on NAEP in Alabama have declined eight points, to below the national average. Grade 8 reading scores remain stagnant.

I don’t like to made pronouncements about policy issues without talking to a range of people and looking into the details of what changes could be made. But my working hypothesis would be that the reading initiative needs to be restored.

STEM (science, technology, engineering, and mathematics) improvement. Alabama also had a long-running math, science, and technology initiative. In contrast with the reading initiative, it seems to have produced no noticeable effects on student achievement. I think it makes sense for Alabama to look into why there has been such a sharp difference in results between the two initiatives, and make appropriate adjustments.

In both STEM and reading, the Alabama State Department of Education should consider certifying Professional Development programs that are based on science – based on empirical research with control groups and the like. A State superintendent should seek to make sure that districts know about good Professional Development and should nurture the growth of such programs.

Graduation rate and academic attainment. To boost academic attainment, Alabama should put in place a 3rd grade test (Florida currently retains students who cannot pass its 3rd grade reading test) and a 10th-grade/graduation test. Alabama discarded its High School Exit Examination (AHSGE – Alabama High School Graduation Exam) in 2013 and replaced it with ACT/CCRS (College- & Career-Ready Standards).

The 11th grade ACT cut scores in Alabama are 18 for English; 22 for math and reading; and 23 for science. The ACT scores are the standard current national “college readiness” scores. So they are, in fact, quite demanding and above the 10th grade level.

ACT is basically used in Alabama for federal reporting purposes to comply with the federal Elementary & Secondary Education Act.  Graduation is currently based on area requirements and seat-time (measured in Carnegie units).

Alabama should restore its exit exam, but it should operate in a multiple second-chances way, with many opportunities to retake the tests and offering a variety of exemptions and special categories.  The idea is to create a focus and a shared goal, not to deny some large number of students a diploma.  Alabama should also reconsider its One Diploma policy.  Since there are a wide variety of students with different capacities, interests, and needs, there should probably be a variety of diplomas.

Tom Loveless of the Brookings Institution estimates that all the prep time and all the millions spent on the Common Core national curriculum-content standards boosted achievement in the U.S. by only a tiny amount: one scale-score point. The new Every Student Succeeds Act does much to protect against future federal interference with curriculum. When it comes to testing, Alabama is not now with PARCC or SmarterBalanced, but has instead affiliated with ACT. Yet Alabama has a set of state standards largely based on Common Core. It is time, I would think, to revisit those current state standards to see what needs fixing. (I participated in California’s line-by-line review of the Common Core in 2010, and I participated in creating the late-1990s California State Academic Content Standards, judged best in the nation by the Thomas B. Fordham Foundation and the American Federation of Teachers.)

Career and technical education. Students, because they are different individuals, should have different educational pathways in high school. Blouke Carus — a leading children’s magazine publisher (including Cricket, Ladybug, Cobblestone), math and reading textbook developer, and chairman emeritus of the Carus Corporation (a company producing manganese compounds) — has said: “Our schools need to offer each student a choice among six or more challenging and rigorous high school curricula, as do other, higher-performing countries.”

As I and my colleagues wrote in a 2013 manifesto entitled “Closing the Door on Innovation”: “There is no evidence to justify a single high school curriculum for all students. A single set of curriculum guidelines, models, or frameworks cannot be justified at the high school level, given the diversity of interests, talents, and pedagogical needs among adolescents. American schools should not be constrained in the diversity of the curricula they offer to students. Other countries offer adolescents a choice of curricula; Finland, for example, offers all students leaving grade 9 the option of attending a three-year general studies high school or a three-year vocational high school, with about 50% of each age cohort enrolling in each type of high school. We worry that the ‘comprehensive’ American high school may have outlived its usefulness, as a [2011] Harvard report [Pathways to Prosperity] implies.”

Because vocational education has a reputation for being a dumping ground and a place where minority children were channeled because of racial bias, education policy has neglected vocational education and sometimes even eliminated it. Alabama’s “Essentials/Life Skills Pathway” is a separate vocational-education pathway.  For example, instead of English 9, the student would take English Essentials 9. This is intended to prep students for community college or the workforce, not for entry to 4-year colleges.

Alabama needs to avoid a situation which the only high school available for a student is a college-prep comprehensive or, alternatively, vocational-only.  The state should incentivize local districts (through waivers) to create a variety of career-tech program alternatives (including a variety of mission-oriented schools), but these should include an academic component (as they do now). In general, Alabama should look carefully at Arizona where the process of school creation is highly decentralized, and the results have been fruitful.

Teacher Quality. As the 1999 manifesto “The Teachers We Need” (of which I was a co-signer) says, “the surest route to quality is to widen the entryway [and] deregulate the processes.” This would expand the pool of potential teachers. It would open the doors to highly-qualified, but non-traditional recruits who want to become teachers and thereby improve overall quality. New York State currently has a relatively open process allowing alternative certification.  In Arkansas, almost half the incoming teachers come in under alternative certification.

But Alabama gets only a C- from the National Council on Teacher Quality for the extent to which the state’s teaching pool has been expanded by alternative certification and the like. Alabama should also be open to non-traditional teacher training programs like the Relay Graduation School of Education in New York City.

Michael McShane of the American Enterprise Institute has written:

“Recently, I have been influenced by the work of Northwestern University economist Kirabo Jackson, whose fascinating NBER working paper calls into serious question policy’s recent overreliance on math and reading scores as the primary measure of the ‘goodness’ of schools and teachers. As it turns out, teachers have important and measurable impacts on both the cognitive and non-cognitive development of students. While it’s certainly true that test scores can tell us something important about a teacher, what is troubling for the test-score types is that it looks like (1) non-cognitive scores are better predictors of later life success (completing high school, taking the SAT, and going to college) and (2) that it is not the same set of teachers that is good at raising both cognitive and non-cognitive measures.

“Such has to be the same for schools, right? If there are teachers that are increasing non-cognitive, but not cognitive skills, surely there are schools that are doing the same. As a result, trying to assess if a school is ‘good’ or ‘bad’ relies on a complex web of preferences and objective measures that, quite frankly, cannot be taken into account in a centralized accountability system. We need something more sophisticated, and something that can respect a diverse conception of what ‘good’ and ‘bad’ means.”

In citing Michael McShane and Kirabo Jackson, I am suggesting a goal for the future, while not saying that there should be no accountability. We should consider a change in providers of accountability because Alabama has a variety of school districts, localities, and other groupings. As Jason Bedrick of the Cato Institute and Andrew Smarick, chair of the Maryland State Board of Education, have separately pointed out, accountability doesn’t have to be in conformity with One Best Way. School evaluation can be pluralistic and rivalrous and doesn’t have to be solely governmental. Churches could do it, neighborhood associations could do it, chambers of commerce could do it.   As Bedrick writes: “Parents can then evaluate the quality of education providers based on their own experience and the expert evaluations of appropriate external providers, and the entire system evolves as parents select the providers that best meet their children’s needs.”

Currently if a teacher is educated in Alabama teachers colleges and receives bad evaluations during the first three years, the teacher can go back to college for free. This seems like a smart program worth retaining.

Education Next magazine surveyed teachers nationwide and the teachers themselves said that about 5% of teachers deserve an “F” and 8% deserve a “D.” But we should be cautious about grandiose projects to improve teacher quality. The “highly qualified teacher” provisions of the No Child Left Behind law were never successfully put into effect. The elaborate new teacher-evaluation systems put in place (at the cost of millions) in about 20 states have reduced the percentage of teachers rated “satisfactory” from 99% to 97%.

Currently teacher-certification testing is under court supervision. Alabama has low passing scores on the PRAXIS II tests that it expects of teachers. When court supervision is over, the state could slowly raise passing scores (as Texas has). Teachers do need to know the subject-matter they are teaching. But the most important thing is to increase the inflow of good prospects into the teaching pool (from which districts and schools can select), rather than reducing the size of the pool.

Economic growth and student attainment. Both policymakers and the civically active public underestimate the economic gains from school improvement. The differences in rates of growth among states can be matched to the education of the workers in the various states. A multiplicity of causes affect economic growth, but nothing is more influential in the long run than school improvement. Although it is certainly difficult to improve schools, it is easier to improve schools than it is to change other factors that figure in increased productivity.

Equated to PISA international test results, Alabama is currently below Turkey.

Some people think that Alabama’s low results are because of demographics. But the only state whose whites do worse currently is West Virginia.

For Alabama, which is educationally at the bottom of the lowest quartile in the United States, the present value of bringing the state to the level of Kentucky – in the next quartile — would be three times Alabama’s current Gross Domestic Product (according to my Hoover Institution colleague Eric A. Hanushek). Other states at the Kentucky level are Florida, Arizona, Oklahoma, and Rhode Island. To get to the level of Kentucky would mean for Alabama over 6% annual growth in GDP and about 12% annual growth in salaries.

Conclusion: A State Superintendent should not be a Caesar, lording it over local districts and local superintendents. State superintendents should not act as if they are progressive technocrats with coercive utopian powers. It will be harmful to learning communities if state superintendents act that way.  Where sensible, they should devolve much of the responsibility over academic content, teaching methods, and instructional materials to the local districts. The different districts can – and should — try different things. Parallel learning communities will sometimes arise.  If districts and schools endeavor to conscientiously do their best, Alabama can climb out of the cellar and surpass states like Kentucky academically. The State of Alabama will prosper, and Alabama’s children will have more fulfilling lives.

—————————————————————

Williamson M. Evers, Ph.D., is a Research Fellow at Stanford University’s Hoover Institution and a former U.S. Assistant Secretary of Education for Planning, Evaluation, and Policy Development. He is a finalist for the position of Alabama State Superintendent of Education.


Random Thoughts on the Passing Ed Policy Scene

May 13, 2016

deep-thoughts-hulu

(Guest Post by Jason Bedrick)

Every now and then, Thomas Sowell writes a column titled “Random Thoughts on the Passing Scene” where he offers up gems like this:

Stupid people can cause problems, but it usually takes brilliant people to create a real catastrophe.

I’m no Thomas Sowell, but here are a few of my own (much less pithy or clever) random thoughts the passing education policy scene:

Montana Department of Revenue: Religious Families Need Not Apply

Last October, the Montana Department of Revenue interpreted the state’s constitution to prohibit it from issuing tax-credit scholarships to students attending religious schools. Fortunately, a judge ordered them to reverse course after the heroes at the Institute for Justice — which deserves its title as the “nation’s pre-eminent courtroom defender of school choice” — filed suit. The injunction is only temporary, pending the outcome of the case, but the MT-DOR bureaucrats just filed an appeal. Can’t let parents get a taste of choice! Why, they might choose something else!

Sadly, they’re far from the only ones working to block educational choice…

The Left v. Educational Choice

Lawsuits against two of the most ambitious educational choice policies are each one step closer to resolution. In Florida, the teachers’ union (joined by the NAACP and others) is challenging the state’s tax-credit scholarship program, which served nearly 70,000 low-income students. A lower court tossed out the challenge based on standing but the union appealed. Earlier this week, an appeals court heard oral arguments in the case, where the union’s attorney struggled to explain how the choice law harms anyone:

Lynn Hearn, arguing for the groups challenging the program, said students receiving scholarships are spread unevenly across the state. The program affects public schools “by drawing students out of the system and sending the funding away,” she said, and schools don’t always reap savings from having fewer children to educate.

“You’ll have a few from this school and a few from that school, and so the school is left with exactly the same expenses,” she said.

Judge Lori Rowe, part of a three-judge panel that heard the case, was skeptical.

“You haven’t alleged that any individual student is suffering,” she said to Hearn. “You haven’t alleged that per-student funding has been reduced. You haven’t even alleged that the [state] education budget has been reduced.”

Meanwhile, the Nevada Supreme Court is preparing to hear oral arguments in a lawsuit challenging the state’s education savings account law. Last month, the attorney general’s office “formally nudged” a judge to rule on a separate challenge by the ACLU that was filed back in August. There’s a decent chance that the state supreme court will merge the two cases.

Why is the left so determined to keep families from exercising choice? In short, it’s about control. And that leads me to…

The Bathroom Wars

It’s frankly incredible that with all the serious problems this country is facing, the president thinks it’s a priority to issue a decree forcing public schools to let children pick which bathroom–or locker room–they want to use. Yes, that’s right, the feds have ordered public schools to give biological males access to the girls’ locker room, where teenage girls undress and shower. The White House press secretary, the Orwellianly named Josh Earnest, claims that the new decree is only “guidance” not a “threat,” but if schools don’t follow the “guidance” then the Obama administration promises to cut their federal funding.

Late last year, the feds intervened when a public school in Illinois required a transgender student who is biologically male to “change and shower separately from her teammates and classmates.” Rather than force the student to use the boys’ locker room, the school came to a compromise that attempted to respect the privacy of all the students involved. Nevertheless, the feds filed suit, demanding the transgender student receive “unfettered access” to the girls’ locker room, even though some teenage girls expressed discomfort undressing and showering with someone who is anatomically male. Eventually, the school caved. Outraged parents have filed a lawsuit seeking to overturn the school’s agreement with the feds, and this week the ACLU announced it was intervening to support the feds.

As Neal McCluskey (my wise and benevolent boss at the Cato Institute) often observes, the zero-sum nature of political control over schooling forces citizens into social conflicts like the Bathroom Wars. A much less divisive alternative would be (you guessed it!) a system of educational choice in which parents could select the schools that have their preferred bathroom policy:

As important, if not more so, is that allowing private entities to choose their own policies is consistent with individual liberty, including freedom of association and religion, while it is much better suited to enabling people with competing values to peacefully co-exist. There is no zero-sum contest: Those who want an open bathroom policy could choose schools in which all the staff and families also embraced it, while those feeling more comfortable with bathrooms and locker rooms restricted by biological sex could go to schools with like-minded people.

Perhaps the best examples of educational choice helping to bring peace and balance rights have been in many European countries, where religious conflicts in schools abated as governments decided to fund choices of Protestant, Catholic, nonsectarian, or other institutions. Getting to the place of greater peace requires something difficult – accepting that all people should be able to live as they want as long as they do not force themselves on others, and even if we do not like the choices they make – but living and letting live is the foundation of a free society.

But forget bathrooms–schools might want to consider separating the genders in the classroom as well…

New Study: Single-Sex Schooling Works

A new study by Prof. C. Kirabo Jackson of Northwestern University finds that single-sex schooling improves student outcomes:

The results show that single-sex education can improve both boys’ and girls’ outcomes. Three years after being assigned to a single-sex secondary school, both boys and girls have higher scores on standardized tests. Five years later, they are more likely to take and pass advanced courses. In the long run, both boys and girls are more likely to have completed secondary school and to have earned the credential required to continue to tertiary education. Importantly, boys are also less likely to have been arrested. Taken as a whole, the results suggest that being in the single-sex cohorts improved test scores and also improved longer-run non test score outcomes such as advanced course taking, high school completion and engaging in criminal activity.

As AEI’s Michael Strain highlighted, “the benefits of single-sex instruction are free to the taxpayer — all you have to do is sort children into the appropriate classrooms or schools.”

You won’t be surprised to learn that, evidence be damned, the ACLU opposes this as well.


The weak predictive power of test scores

May 2, 2016

Here’s my first round in the debate with Mike Petrilli over whether test scores are reliable indicators of quality that can be used by regulators and policymakers to identify schools to be closed or expanded…

——————————————————-

The school choice tent is much bigger than it used to be. Politicians and policy wonks across the ideological spectrum have embraced the principle that parents should get to choose their children’s schools and local districts should not have a monopoly on school supply.

But within this big tent there are big arguments about the best way to promote school quality. Some want all schools to take the same tough tests and all low-performing schools (those that fail to show individual student growth over time) to be shut down (or, in a voucher system, to be kicked out of the program). Others want to let the market work to promote quality and resist policies that amount to second-guessing parents.

In the following debate, Jay Greene of the University of Arkansas’s Department of Education Reform and Mike Petrilli of the Thomas B. Fordham Institute explore areas of agreement and disagreement around this issue of school choice and school quality. In particular, they address the question: Are math and reading test results strong enough indicators of school quality that regulators can rely on them to determine which schools should be closed and which should be expanded—even if parental demand is inconsistent with test results?

To a very large degree, education reform initiatives hinge on the belief that short term changes in reading and math achievement test results are strong predictors of long term success for students. We use reading and math test scores to judge the quality of teachers, schools, and the full array of pedagogical, curricular, and policy interventions. Math and reading test scores are the yardstick by which education reform is measured. But how good of a yardstick is it?

Despite the centrality of test scores, there is surprisingly little rigorous research linking them to the long-term outcomes we actually care about. The study by researchers from Harvard and Columbia (Chetty, et al.) showing that teachers who increase test scores improve the later-life earnings of their students is a notable exception to the dearth of evidence on this key assumption of most reform initiatives. But that is one study, it has received some methodological criticism (although I think that has been addressed to most people’s satisfaction), and its results from low-stakes testing may not apply to the high-stakes purposes for which we would now like to use them. This seems like a very thin reed on which to rest the entire education reform movement.

In addition, we have a growing body of rigorous research showing a disconnect between improving test scores and improving later-life outcomes. I’ve written about this at greater length elsewhere (see here and here), but we have eight rigorous studies of school choice programs in which the long-term outcomes of those policies do not align with their short-term achievement test results. In four studies, charter school programs that produce impressive test score gains appear to yield no or little improvement in educational attainment. In three studies of private school choice and one charter school choice program, we observe large benefits in educational attainment and even earnings but little or no gains in short-term test score measures.

If policy analysts and the portfolio managers, regulators, and other policy makers they advise were to rely primarily on test scores when deciding which programs or schools to shutter and which to expand, they would make some horrible mistakes. Even if we ignore the fact that most portfolio managers, regulators, and other policy makers rely on the level of test scores (rather than gains) to gauge quality, math and reading achievement results are not particularly reliable indicators of whether teachers, schools, and programs are improving later-life outcomes for students.

What explains this disconnect between math and reading test score gains and later-life outcomes? First, achievement tests are only designed to capture a portion of what our education system hopes to accomplish. In particular, they are not designed to measure character or non-cognitive skills. A growing body of research is demonstrating that character skills like conscientiousness, perseverance, and grit are important predictors of later-life success (see this, for example). And more recent research by Matt KraftKirabo Jackson, and Albert Cheng and Gema Zamarro (among others) shows that teachers, schools, and programs that increase character skills are not necessarily the same as those that increase achievement test results. There are important dimensions of teacher, school, and program quality that are not captured by achievement test results. Second, math and reading achievement tests are not designed to capture what we expect students to learn in other subjects, such as science, history, and art. Prioritizing math and reading at the expense of other subjects that may be important for students’ later-life success would undermine the predictive power of those math and reading results. Third, many schools are developing strategies for goosing math and reading test scores in ways that may not contribute to (and may even undermine) later-life success. The fact that math and reading achievement results are overly narrow and easily distorted makes them particularly poor indicators of quality and weak predictors of later-life outcomes.

I do not mean to suggest that math and reading test results provide us with no information or that we should do away with them. I’m simply arguing that these tests are much less reliable indicators of quality than most policy analysts, regulators, and policy makers imagine. We should be considerably more humble about claiming to know which teachers, schools, and programs are good or bad based on an examination of their test scores. If parents think that certain teachers, schools, and programs are good because there is a waiting list demanding them, we should be very cautious about declaring that they are mistaken based on an examination of test scores. Even poorly educated parents may have much more information about quality than analysts and regulators sitting in their offices looking at spreadsheets of test scores.

I also do not mean to suggest that policy makers should never close a school or shutter a program in the face of parental demand. I’m just arguing that it should take a lot more than “bad” test scores to do that. Yes, parents can and will make mistakes. But analysts, authorizers, regulators, and other policy makers also make mistakes, especially if they rely predominantly on test results that are, at best, weak predictors of later-life success. The bar should be high before we are convinced that the parents are mistaken rather than the regulators poorly guided by test scores. Besides, we should prefer letting parents make mistakes for their own children over distant bureaucrats making mistakes for hundreds or thousands of children while claiming to protect them.

(Also posted at Flypaper )


The Wisdom of the Market

July 17, 2015

Design vs Experience

(Guest Post by Jason Bedrick)

I’m humbled that Andy Smarick thinks I “offered the most compelling philosophical explanation for a system of choice” among the recent Fordham Institute Wonkathon participants. However, he misreads me when he states that my “professed ‘humility’ (we don’t know everything) came across as agnosticism (we can’t know anything) given that we’ve learned gobs about choice over twenty-five years.”

Nowhere do I claim “we can’t know anything.” Of course we do, and of course we can learn more. But questions remain: who are “we,” what do “we” know, and is that knowledge sufficient to achieve “our” ends?

By “we” Smarick seems to have in mind “policymakers and wonks” and the “what” that “we” supposedly know is that markets alone just don’t cut it so some very, very smart people must bend schools to their will impose government regulations to ensure accountability. To bolster his case, Smarick cites a recent article in National Affairs by Chester Finn and Bruno Manno on the lessons they’ve learned from their decades of experience studying charter schools:

Both strongly support school choice, but they concede the “vexing reality” that “market forces alone can’t reliably generate academic effectiveness.” Overconfidence led to accountability getting short shrift early. “Those present at the creation of the charter bargain (ourselves included) paid too little attention to how authorizing would work.”

Throughout the article, the [Finn and Manno] explain how events played out differently than expected. Because they assumed “a barely regulated marketplace would provide more quality control than it has…we focused on quantity rather than quality.” They were excited by policies that would spur “infusions of capital and entrepreneurialism,” but “we didn’t take seriously enough the risk of profiteering.”

Smarick claims that my Wonkathon entry’s “sanguine title, ‘Let the market work,’ runs headlong into Finn and Manno’s reflections.” As Justice Scalia might say: pure applesauce.

It may or may not be true that “market forces alone can’t reliably generate academic effectiveness” but Finn and Manno cannot draw that conclusion from experience in the charter sector because charters are not operating in a free market, never mind a “barely regulated” one. Charters are secular public schools that can’t charge families tuition, can’t devise their own criteria for admission, they have to meet certain state standards, and they can be shut down by their “authorizers” even if a sufficient number of students and parents prefer the charter to make it financially viable.

In other words, charters provide more choice and competition than the status quo, but charters are not operating in a market. The lack of a price mechanism alone should make that apparent. Drawing any conclusions about what an actual market in education would or would not produce based on the charter experience is ludicrous.

There’s some truth to Smarick’s contention that “theory without experience is [mere] intellectual play,” but he’s drawing the wrong lessons from Finn and Manno’s experience. Although it’s impossible to draw solid conclusions about a market from a non-market, the charter sector has much to teach policymakers about the chasm between policy intentions and policy results.

For example, Finn and Manno lament that “charters in many places are hobbled by many operational constraints, too little money, and, often, insufficient attention both to the delicate balance between quantity and quality.” These constraints often stem from the very regulatory framework that was intended to ensure quality. A 2010 Fordham study found that “state laws were the primary sources of constraint on charter school autonomy, accounting for three-quarters of the infringement that these schools experience.” This year, an American Enterprise Institute study found that “one-fourth of average charter application contains inappropriate and onerous requirements,” and that authorizes “sometimes mistake length for rigor” and “often prize innovation less than they say they do.”

The fatal conceit of the charter school agenda was that granting schools a bit more autonomy and granting parents a bit more choice in a controlled environment would create a true “market” in education. But a market requires a price mechanism, a means of channeling dispersed knowledge. Smarick accuses me of believing that “we can’t know anything” but that’s not so. Policymakers can’t possibly ever know enough to design the education system from the top down but the market can channel dispersed knowledge to produce higher quality through experimentation, evaluation, and evolution.

Smarick confuses the technical knowledge of experts for the dispersed knowledge of the entire system. Sure, technocrats have learned “gobs about choice” in a quarter-century, but they can’t possibly know enough to design the most effective possible education system. Likewise, a panel of a dozen Nobel-prize winning economists certainly knows “gobs” about how markets function, but they cannot possibly know enough to effectively set the price of tin on any given day.

The technocrats’ approach is attempt to define quality, measure it, and shut down any school that doesn’t live up to it — even against the will of parents. As Finn and Manno wrote:

Charter doctrine is clear: Bad schools should be closed (or “non-renewed”) by their authorizer. Yet it turns out to be as hard to close bad charters as traditional district schools. Hundreds of kids are affected, and the alternatives for them are often no better than the troubled charter. Furthermore, parents are almost universally hostile to the demise of their children’s current schools.

First, why are those parents “almost universally hostile” to closing down their chosen school? Could it be because “the alternatives for them are often no better” and probably worse? Could it be that the schools are effectively providing some things–safety, discipline, good values, a love of learning–that the parents legitimately prioritize over test scores in a few subjects?

Attempts to define and measure quality too often come at the cost of innovation. At present, states’ standardized testing regimes assume that all students should progress at the same pace across all subjects — a system that is anathema to reforms like competency-based learning which dispense with Carnegie units. Moreover, the focus on a few subjects both creates a perverse incentive for schools to focus on those subjects to the exclusion of others and overlooks the other, often more important areas where a school may be performing quite well. As AEI’s Michael McShane wrote:

Recently, I have been influenced by the work of Northwestern University economist Kirabo Jackson, whose fascinating NBER working paper calls into serious question policy’s recent overreliance on math and reading scores as the primary measure of the “goodness” of schools and teachers. As it turns out, teachers have important and measurable impacts on both the cognitive and non-cognitive development of students. While it’s certainly true that test scores can tell us something important about a teacher, what is troubling for the test-score types is that it looks like (1) non-cognitive scores are better predictors of later life success (completing high school, taking the SAT, and going to college) and (2) that it is not the same set of teachers that is good at raising both cognitive and non-cognitive measures.

Such has to be the same for schools, right? If there are teachers that are increasing non-cognitive, but not cognitive skills, surely there are schools that are doing the same. As a result, trying to assess if a school is “good” or “bad” relies on a complex web of preferences and objective measures that, quite frankly, cannot be taken into account in a centralized accountability system. We need something more sophisticated, and something that can respect a diverse conception of what “good” and “bad” means.

This is not to say that there should be no standards or accountability. The question is who imposes the accountability on whom. As I’ve noted previously, the absence of a government-imposed standard does not imply the lack of any standards. Rather, it leaves space for competing standards, which in turn fosters innovation and diversity. Parents can then evaluate the quality of education providers based on their own experience and the expert evaluations of appropriate external providers, and the entire system evolves as parents select the providers that best meet their children’s needs.

So yes, policymakers should be humble about what they know or think they know, but we can have greater confidence in a system that channels dispersed knowledge to produce greater quality and innovation. This is more than mere “intellectual play.” It’s the process by which we’ve seen enormous gains in productivity and quality in nearly every other sector in the last century — not top-down technocratic tinkering but bottom-up experimentation, evaluation, and evolution in a free market.


Does Money Matter After All?

July 7, 2015

(Guest Post by Eric A. Hanushek)

Considerable prior research has failed to find a consistent relationship between school spending and student performance, making skepticism about such a relationship the conventional wisdom. Given that skepticism, new studies that purport to find a systematic relationship between school spending and student performance get disproportionate attention.

There is in fact great demand for results linking funding with favorable outcomes.  Knowing that a strong relationship existed would mean that policy makers outside of the schools – legislatures, governors, and courts – only have to concern themselves with how much money was provided to schools and not with how money was used.  And, meeting our education challenges by providing more money appears from history to be easier than pursuing more fundamental changes in schools.

Kirabo Jackson, Rucker C. Johnson, and Claudia Persico offer a new study suggesting that a clear money-performance relationship exists if you just look in the right place. Their overarching conclusion is that “methods matter.” Their discovery of a money-performance relationship is attributed to analyzing the effects of spending that emanates from court decisions (exogenous variation in spending), tracing the effect of this spending to long run outcomes (completed schooling and wages), and focusing on the right subgroup (disadvantaged students).

From a methodological viewpoint, details are important here. How court decisions are dated given long and repeated legal involvement in many states; how the spending reaction to court decisions is measured; whether the court decisions are unrelated to the character of schools before court involvement; and how court-mandated spending differs from other increased spending are a few of the details.  Nevertheless, while these are important methodological issues, it is more useful to focus on the substance of their findings.

Jackson, Johnson, and Persico reach the following conclusions about the impact of a 10 percent increase in spending for all 12 years of schooling:  1) It would increase years of schooling by 0.44 years for poor children and by an insignificant 0.075 years for non-poor children, implying that a spending increase of 22.7 percent would eliminate the average education gap; 2) It would increase high school graduation rates by 11.6 percentage points for poor children and 6 percentage points for non-poor children; 3) It would increase subsequent family incomes by 16.4 percent for poor children and zero for non-poor children; 4) It would reduce subsequent adult poverty rates by 6.8 percentage points for poor children and zero for non-poor children.

Their analysis covers schooling experiences for the period 1970-2010.  Thus, it is useful to connect these estimates to actual funding patterns over the period.  Between 1970 and 1990, real expenditure per pupil increased not by 10 percent but by over 84 percent.  By 2000, this comparison with 1970 topped 100 percent, and it reached almost 150 percent by 2010.  No amount of adjustment for special education, LEP, or what have you will make these extraordinary increases in school funding go away.

If a ten percent increase yields the results calculated by Jackson, Johnson, and Persico, shouldn’t we have found all gaps gone (and even reversed) by now due to the actual funding increases?  And, even with small effects on the non-poor, shouldn’t we have seen fairly dramatic improvements in overall educational and labor market outcomes? In reality, in the face of dramatic past increases in school funding, the gaps in attainment, high school graduation, and family poverty have remained significant, largely resisting any major improvement.  And, the stagnating labor market performance for broad swaths of the population has captured considerable recent public and scholarly attention.

What could reconcile these apparent inconsistencies?  Here are some possibilities:

  • There might be sharply diminishing returns to spending so that their estimates apply most clearly in 1970 when spending (in 2011-12 $’s) was just $4,500 as opposed to 2010 when spending was $11,000. Thus, by implication, spending today might be expected to have a much smaller impact than they estimate.
  • Only spending induced by the courts might have the large impacts they identify, with spending not related to judicial rulings having a negligible impact. The likelihood of this is a little questionable since, with a few exceptions such as NJ and WY, the courts have done little to look into how any legislature responds in terms of specific spending programs.  But, if true, normal spending increases by state legislatures and by local taxpayers would not be expected to have any impact on outcomes.
  • The estimates of Jackson, Johnson, and Persico might simply be very wrong.

Maybe there are other ways to reconcile the Jackson, Johnson, and Persico estimates with the aggregate data on spending and outcomes.  But in the end they themselves state what is now commonly accepted: How money is spent matters.  Indeed, by simple consideration of their evidence, how money is spent is more important than how much is spent.

Of course, it is always important to recognize that none of this discussion suggests that money never matters.  Or that money cannot matter.  It just says that the outcomes observed over the past half century – no matter how massaged – do not suggest that just throwing money at schools is likely to be a policy that solves the significant U.S. schooling problems seen in the levels and distribution of outcomes.  We really cannot get around the necessity of focusing on how money is spent on schools.


Does School Spending Matter After All?

May 29, 2015

This is the question raised by a new study by C. Kirabo Jackson, Rucker C. Johnson and Claudia Persico  in Education Next.  Jackson, et al claim to have up-ended decades of school finance research by finding a link between school spending and improved student outcomes.  After reading that article and an earlier, more detailed version posted on the NBER web site, I find nothing to persuade me to abandon the long-standing and well-established finding that simply providing schools with more resources does not improve student outcomes.

Let’s remember how well-established this finding is by noting that Eric Hanushek conducted a comprehensive review of the literature and concluded:

…the research indicates little consistent relationship between resources to schools and student achievement. Much of the research considers how resources affect student achievement as measured by standardized test scores. These scores are strongly related to individual incomes and to national economic performance, making them a good proxy for longer run economic impacts. But, the evidence – whether from aggregate school outcomes, econometric investigations, or a variety of experimental or quasiexperimental approaches – suggests that pure resource policies that do not change incentives are unlikely to be effective. (p. 866)

Jackson, et al acknowledge that past research has failed to find a link between school resources and student outcomes:

Coleman found that variation in school resources (as measured by per-pupil spending and student-to-teacher ratios) was unrelated to variation in student achievement on standardized tests. In the decades following the release of the Coleman Report, the effect of school spending on student academic performance was studied extensively, and Coleman’s conclusion was widely upheld.

But they believe that past research was flawed in two important respects.  First, test scores may be a weak indicator of later-life success, so it would be better to look at stronger measures, like educational attainment, employment, and earnings.  Second, they believe that past studies of school spending may suffer from an endogeneity problem.  That is, extra money has tended to go to schools facing challenges.  The failure to find a link between more resources and better achievement may be because schools with a weaker future trajectory are the ones more likely to get more money.  So, the causal arrow may be going in the wrong direction.  Weak performance may be causing more resources rather than more resources causing weak performance.

Jackson, et al solve the first issue by focusing on longer-term student outcomes, like educational attainment and earnings.  They claim to have a solution to the second problem by finding a type of spending increase that is unrelated to the expected trajectory of school performance.  Court-ordered spending, they say, is exogenous, while regular legislative increases in spending are endogenous.

The surprising findings of the Jackson, et al article hinge entirely on this claim that court-ordered spending is exogenous.  Looking at attainment and earnings by itself does not produce a different result than past research that has focused on test scores.  The thing that allows Jackson, et al to find that spending is linked to better student outcomes is the fact that they do not examine actual spending increases.  Instead, they predict changes in spending based on court-orders and use that predicted spending in place of the actual spending.

This instrumental variable technique developed by James Heckman, however, only works if the instrument is in fact exogenous.  That is, court-ordered spending has to be unrelated to the future trajectory of school performance.  Given how critical this point is to the entire article, you might think Jackson, et al would spend a fair amount of energy to justifying the exogeneity of court-ordered spending.  They do not.

It is completely mysterious to me why we should believe that court-ordered spending differs from legislatively-originated spending in the likelihood that it is linked to the expected future trajectory of school performance.  That is, schools facing challenges are just as likely to get extra money if the spending originates in the courts or in the legislature.  If we are concerned that the causal arrow is going in the wrong direction in that weak performance causes more money rather than the other way around, we should have that concern just as much whether the motivation for the money came from the court or the legislature.

Jackson, et al do not make a proper case for the exogeneity of court-ordered spending other than to describe it as a “shock” to school spending.  But there is nothing more shocking about spending that originates in the courts than in the legislature.  Court cases take years to develop, be decided, and complete appeals.  And then they have to be implemented by legislative action.  The timing of court-ordered spending is no more surprising to schools than regular legislative spending.  Nor is the amount of spending change necessarily more dramatic than those originating in legislatures.  The passage of ESEA and its re-authorizations infused large amounts of money into schools.

Jackson et al need to convince us that court-ordered spending is exogenous to get their unusual result.  If they just used conventional methods, they would confirm the wide-spread finding that extra money does not improve outcomes.  As they describe it:

We confirm that our approach generates significantly different results than those that use observed increases in school spending, by comparing our results to those we would have obtained had we used actual rather than predicted increases as our measure of changes in district spending. For all outcomes, the results based simply on observed increases in school spending are orders of magnitude smaller than our estimates based on predicted SFR-induced spending increases, and most are statistically insignificant.

But Jackson, et al fail to justify the claim that court-ordered spending is exogneous on which their entire article depends nor does such a claim seem plausible.

But even if you were to somehow believe that court-ordered spending is exogenous, it would still be unwise to jump to the conclusion that we now know money matters and should open the resource spigots to K-12 education.  First, the past research Hanushek reviewed includes studies that do not suffer from either of the concerns raised by Jackson, et al.  That is, some of those studies examine later-life outcomes for students and not just test scores and some of those studies rely on experimental methods with which there is no problem with causation.  Why should we disregard those studies for this one new study even if we were to ignore the concerns I’ve raised above?

Second, Jackson, et al are examining the effect of court-ordered spending in the 1970s when spending levels in real terms were much lower and variation in spending across districts within states was much higher.  It’s quite a leap to think that more money now would have the same effect as then.  To my surprise, Bruce Baker made this same point in response to the Jackson, et al article in comments to Education Week:

“[E]xploring such [far-apart] outcomes, while a fun academic exercise, is of limited use for informing policy,” he wrote in an email to Education Week. “Among other things, these are changes that occurred under very different conditions than today.”

Mr. Baker also disagreed with the researchers’ caveat that similar changes might have a much smaller effect if introduced today, in part because total school funding nationwide increased by 175 percent over 43 years, from an average of $4,612 per student in 1967 to about $12,772 per student in 2010, as measured in 2012 dollars.

So does school spending matter after all?  I think the answer is still clearly “no.”


The Other (More Important) Value-Added Measure

February 26, 2013

(Guest Post by Collin Hitt)

Some teachers are better than others when it comes to raising test scores, which in turn can raise students’ earnings in adulthood. But test scores aren’t everything. A new study looks at whether individual teachers can have similar impacts on suspension rates, school attendance, GPA and even graduation rates. It finds that they can, and do.

To put the non-test score estimates into perspective, having an Algebra or English teacher 20 at the 85th percentile of GPA quality versus one at the 15th percentile would be associated with 0.09 and 0.054 higher GPA, respectively. For both subjects, a teacher at the 85 percentile of ontime grade progression quality versus one at the 15th percentile would be associated with being 5 percentage points (0.14σ) more likely to enroll in 10th grade on time. Given that not enrolling in 10th grade is a strong predictor of dropout, this suggests significant teacher effects on dropout…

That’s from Northwestern’s Kirabo Jackson, His working paper uses state-of-the-art value-added methods to identify North Carolina high school teachers who have significant impacts on test scores. He then uses the same methods to see which teachers have an impact on “non-cognitive” behaviors. One would expect – at least, I expected – that the teachers who raise test scores also raise non-cognitive outcomes. Not so.

For all outcomes, Algebra teachers with higher test score value-added are associated with better non-test score outcomes, but the relationships are weak…This indicates that while teachers who raise test score may also be associated with better non-test-score outcomes, most of effects on non-test score outcomes are unrelated to effects on test scores. The results for cognitive ability are consistent with this…

Results for English teachers follow a similar pattern. English teacher effects on English test scores explain little of the estimated effects on non-test score outcomes…

Because variability in outcomes associated with individual teachers that is unexplained by test scores is not just noise, but is systematically associated with their ability to improve typically unmeasured non-cognitive skills, classifying teachers based on their test score value added will likely lead to large shares of excellent teachers being deemed poor and vice versa.

So teachers can have large effects on matters that are supposedly out of their hands. Suspensions, absence rates and GPA are functions of a student’s conscientiousness – or more generally, of his character. This study delivers another blow to the cop out lobby.

But it also presents a huge challenge to the proponents of test-based teacher policies. Read this line again: “classifying teachers based on their test score value added will likely lead to large shares of excellent teachers being deemed poor and vice versa.”  This is not a trivial matter. Jackson shows that non-cognitive outcomes are more strongly correlated with life outcomes than are test scores – especially for students with limited cognitive ability. This paper cannot be ignored.

Hat Tip: Joanne Jacobs

[Edited to correct formatting error and typos]


The Upward Surge of Mankind?

October 30, 2008

(Guest Post by Matthew Ladner)

Florida tripled the number of Hispanic and African American students passing one or more AP exams with a program that included a financial incentive for schools and teachers.

Meanwhile, C. Kirabo Jackson finds positive results for a similar Texas pilot program in Education Next:

According to my assessment, the incentives produce meaningful increases in participation in the AP program and improvements in other critical education outcomes. Establishment of APIP results in a 30 percent increase in the number of students scoring above 1100 on the SAT or above 24 on the ACT, and an 8 percent increase in the number of students at a high school who enroll in a college or university in Texas. My evidence suggests that these outcomes are likely the result of stronger encouragement from teachers and guidance counselors to enroll in AP courses, better information provided to students, and changes in teacher and peer norms.

Gordon Gekko for Secretary of Education? I can see the confirmation hearing speech in my head:

The point is, ladies and gentleman, that greed — for lack of a better word — is good.

Greed is right.

Greed works.

Greed clarifies, cuts through, and captures the essence of the evolutionary spirit.

Greed, in all of its forms — greed for life, for money, for love, knowledge — has marked the upward surge of mankind.

And greed — you mark my words — will not only save public education, but that other malfunctioning corporation called the USA.

Just kidding, but I will say this: we need to continue experimenting with programs like this. They certainly seem to beat throwing money at schools in the hope that they will improve.