Jay P. Greene's Blog

Rick Hess has a thoughtful post today on last week’s dust-up over whether choice schools should be required to take state tests.  Rick is generally sympathetic with the arguments I was making but raises two objections.

First, Rick worries about whether I (and others) are being consistent in opposing testing requirements for choice schools while having “long slammed districts and promoted school choice by pointing to reading and math scores.”  He continues, “I’ve got a lot of sympathy for those who feel like Greene’s position constitutes something of a bait-and-switch, with choice advocates are changing the rules when it suits them.”

Second, Rick thinks there is an inconsistency in  my suspicion that test-prep and manipulation are largely responsible for test score improvements by Milwaukee choice schools after they were required to take high-stakes tests,  while I interpret research from Florida as showing schools made exceptional test score gains when faced with the prospect of having vouchers offered to their students if scores did not improve.  Why would I believe the former is an artifact of test prep, but not the latter?

Let me deal first with Rick’s second objection because it is easier and quicker to address.  I was concerned about whether test prep and manipulation were responsible for the exceptional gains made by low-graded schools that faced the prospects of voucher competition if their results did not improve.  So, Marcus Winters and I examined results from the Stanford-9, a nationally normed low-stakes test, as well as the state’s high-stakes FCAT, to see if the results were similar.  Here is what we wrote:

Schools are not held accountable for their students’ performance on the Stanford-9. As a result, they have little incentive to manipulate the results by “teaching to the test” or through outright cheating. Thus, if gains are witnessed on both the FCAT and the Stanford-9, we can be reasonably confident that the gains reflect genuine improvements in student learning.

The results were similar, showing exceptional gains on both high and low stakes exams, which gave me confidence that the improvements in FL were real.  In Milwaukee we do not have a similar check on whether learning gains were real after high stakes testing requirements were imposed.  In the absence of a low-stakes check, I’m highly skeptical of whether choice schools suddenly improved in quality when they were required to administer the high-stakes tests that the study subjects had been taking all along with lower results.

Rick’s first point — essentially, that I am being hypocritical in opposing testing for choice schools but not for traditional public schools — requires a more complicated response.  I would be happy opposing state testing requirements for all schools (choice and traditional public) if those schools had some reasonable mechanism for accountability.  Choice schools are accountable without testing requirements because parents can choose whether to send their children (and the resources that follow those students) to those schools or not.  If those schools are not accomplishing what parents want, choice schools have difficulty attracting and retaining students and resources.

Most traditional public schools, however, have no meaningful system of accountability.  They receive students and resources regardless of whether they are accomplishing what families want or not.  If schools are not held accountable by choice, then they have to be accountable by some mechanism.  One way to produce this accountability is to require that they administer state tests and meet certain performance benchmarks.  This type of top-down accountability is far less efficient and comprehensive than choice accountability, but it may have to do in the absence of choice.  But if charter, private, and Tiebout choice were to expand to the point where no school was guaranteed students and revenues regardless of performance, then I’d be fine with getting rid of all testing requirements.

Of course, there would still be plenty of information about schools because most schools in choice systems voluntarily administer tests and report results.  They just choose their own tests, just like how they choose their own standards, curriculum, and pedagogy.  And since tests only capture a tiny portion of what most schools are trying to accomplish, parents would collect information on these other outcomes of education just as consumers collect information on the quality of other complicated services their children receive, including summer camp, piano lessons, babysitters, etc… We don’t have state required testing — or even any testing — for most of these services, so parents rely on reputation, word of mouth, direct observation, and other techniques to collect information and make choices.  No system is perfect and people will make mistakes, but I’d rather that parents make their own mistakes than have bureaucrats impose mistakes upon them.

This skepticism about state testing does represent a shift in my thinking that has been underway for a few years now.  I’m sure someone could dig up an old quote from me embracing top-down accountability in a way that I would not do now.  But I’ve seen more evidence and collected more experience over the last several years that has made me much less enamored of state testing.  I’m convinced that state tests are highly imprecise, very limited in what they cover, subject to test-prep and manipulation, unable to capture the diversity of school goals and circumstances, and seldom used to make intelligent decisions about improving schools.  Simply put, I am no longer a supporter of top-down school accountability regimes.  But until we have expanded choice further, I see no practical alternative to continuing state testing for schools not subject to meaningful choice accountability.

In this week’s debate over the wisdom of requiring choice students to take state tests, three points deserve greater emphasis.  First, testing requirements hurt choice because test results fail to capture most of the benefits produced by choice schools.  As Collin Hitt’s piece persuasively argued, a series of rigorous studies have found large long-term benefits for students able to attend schools of choice even when short-term test results show little or no benefit.  Those studies show that charter and private choice schools cause students to graduate high school and go to college at much higher rates.  Those students go to more competitive universities at much higher rates.  And choice causes those students to enjoy much higher salaries later in life.  But if you only looked at short-term test results for these students you would not have expected the magnitude of these benefits.

One (of the many) problems with imposing testing requirements on schools of choice is that it highlights a measure of performance that grossly under-states the benefits of choice.  Given the precarious political position of choice programs, highlighting a measure that severely under-states performance puts those programs in jeopardy.  I can understand why choice opponents favor testing requirements — since they want ammunition to shut choice down or regulate it into oblivion.  But why would choice supporters favor this?  It’s a huge mistake.

Second, the only piece of evidence that Fordham presents to support the claim that state testing requirements improve performance at choice schools is the finding that scores rose when Milwaukee private choice schools were required to take the high stakes state test.  But as Pat Wolf, one of the authors of that study, noted — the score increase may well be just an artifact of private choice schools deciding to start prepping students for that high-stakes test now that they were required to take it.  In other words, Fordham is confusing real learning increases with test manipulation.  Pat was gently warning Fordham not to misinterpret the results in this way.  Despite that warning, Fordham continues to mis-use this research to make their point.

If Fordham continues to incorrectly cite this bit of evidence to support their point, they are in danger of becoming the Diane Ravitch of think tanks.  Wolf similarly warned Ravitch that she misunderstood the graduation rate component of the Milwaukee voucher study, wrongly claiming that attrition was biasing results to show higher graduation rates for voucher students.  Ravitch did not grasp that the result was based on an intention-to-treat analysis and, if anything, that type of analysis under-states the positive effect of choice on graduation rates.  Ravitch either couldn’t understand this or didn’t care about getting it right, so she continues to repeat this incorrect interpretation of the research to advance her agenda.  In the argument about choice and state testing requirements, Fordham is similarly repeating a faulty interpretation that they’ve been warned is mistaken.

Third, despite the lack of evidence that state testing requirements improve outcomes or ensure quality (as they largely acknowledge in an earlier report, “The Proficiency Illusion”), Mike Petrilli continues to push for them because… well, because we’ve got to do something:

Bad schools happen. They happen in the public sector, the charter sector, and, yes, the private sector. And since education is a “public good” as well as a “private good”—because kids’ lives literally hang in the balance and so does the future of the society whose taxpayers are underwriting these costs—we can’t just look the other way….

But the answer cannot be “let the market figure it out.” Because it hasn’t, and it won’t—and somebody must.

Of course, doing something that is ineffective or counter-productive may be worse than doing nothing.  If state testing requirements don’t necessarily make schools better and fail to capture the bulk of the benefits choice schools are producing, then imposing state testing requirements on choice schools just to do something is a really bad idea.  In an effort to prevent all bad things from happening, Fordham may ensure that more bad things will happen.

Fordham’s argument that we need to do something reminds me of the brilliant song Jason Segel wrote for the fictional band, Aldous Snow and the Infant Sorrow, in the movie Forgetting Sarah Marshall — appropriately titled “We’ve Got to Do Something!”  As he is grabbing a cane from a blind man in the music video Aldous (Russell Brand) sings:

You gotta do something,
We gotta do something,
Sometimes I sit in my room and I don’t know what to do,
but we’ve gotta do something!

…and if I was in Government,
Then I’d Government things much more differentlier,
because it ain’t the best way to government things,

 

UPDATE — On reflection, the Ravitch comparison was too harsh.  Ravitch repeated a factual error even after the error was pointed out to her.  Fordham is repeating an ambiguous finding, not necessarily a factually incorrect one.  One could interpret the test score gain produced by choice schools in Milwaukee after high-stakes testing was required as a test prep artifact or as a real learning gain.  I’m strongly inclined toward the test prep explanation , but the other interpretation is not factually mistaken.  Both the original study and Pat’s post warned Fordham about the ambiguity of this result, yet Fordham continues to cite it as proof without clarification or qualification.  It’s not pulling a Ravitch, but it’s also not good.

(Guest Post by Collin Hitt)

There’s mounting evidence that charter schools decrease dropout rates, increase college attendance rates and improve the quality of colleges that college-bound students attend. But so what if these kids go to college? Do they actually graduate? And if charter schools really have lasting impacts, shouldn’t charter schools actually have an impact on how much money students earn? A new working paper examines these questions and the answer, in a word, is yes.

Kevin Booker, Tim Sass, Brian Gill  and Ron Zimmer have now extended their previous research on charter high schools. (Jay wrote about their research and their clever research design a few years back.) They look at students in Chicago and Florida who attend charter schools in eighth grade, some of whom go on to attend charter high schools and some whom go on to attend district-run high schools.

They find that students who attend charter high schools are more likely to graduate high school, attend college and persist in college. Such findings are extremely important. But the paper is truly novel in that it also examines the labor market outcomes for students. From the study:

In Florida, we also examine data on the subsequent earnings of students in our analytic sample, at a point after they could have earned college degrees. Charter high school attendance is associated with an increase in maximum annual earnings for students between ages 23 and 25 of $2,347—or about 12.7 percent higher earnings than for comparable students who attended a charter middle school but matriculated to a traditional high school.

Two years ago, the front page of the New York Times carried a headline that teachers can have lasting impacts on student’s earnings in adulthood, citing groundbreaking work by Jonah Rockoff, Raj Chetty and John Friedman. For a single school year, a one standard deviation increase in teacher quality – as measured by a teacher’s valued-added impact on test scores – increased a student’s annual earnings at age 28 by $182. Compare that to the impact of attending a charter high school in Florida: a $2,347 increase in annual earnings by age 25. Using Rockoff, Chetty and Friedman’s estimate, that’s equivalent to a student experiencing a one standard deviation in teacher quality every year from kindergarten through the twelfth grade.

So these findings stand out. Moreover, Booker and colleagues close the paper with a key observation. In Florida, as in other school choice research, a paradox became apparent. The improvements in long-term outcomes were in no way predicted by earlier research on test score impacts.

The substantial positive impacts of charter high schools on attainment and earnings are especially striking, given that charter schools in the same jurisdictions have not been shown to have large positive impacts on students’ test scores (Sass, 2006; Zimmer et al., 2012)…

 Positive impacts on long-term attainment outcomes and earnings are, of course, more consequential than outcomes on test scores in school. It is possible that charter schools’ full long-term impacts on their students have been underestimated by studies that examine only test scores. More broadly, the findings suggest that the research examining the efficacy of educational programs should examine a broader array of outcomes than just student achievement.

This, I can promise, will be a recurrent theme in school choice research in the coming years. Recall this passage from Will Dobbie and Roland Fryer’s research of the Harlem Promise Academy, where they found large gains in college attendance:

 “…the cross-sectional correlation between test scores and adult outcomes may understate the true impact of a high quality school, suggesting that high quality schools change more than cognitive ability. Importantly, the return on investment for high-performing charter schools could be much larger than that implied by the short-run test score increases.”

Test scores are supposed to be an indicator of how kids will fare later in life. Now we have another piece of school choice research finding that test scores missed the true positive impact that schools (and choice) had on kids. Something to think about if you’re going to argue that schools of choice should be held more accountable to state tests.