Jay P. Greene's Blog

The problem: too many choices?

(Guest Post by Jason Bedrick)

The abundance of choices among competing grocery stores is “clogging” our roads, according to an Australian academic who advocates assigning people to government-run grocery stores based on the location of their home:

Almost 60 per cent of Melbourne shoppers are bypassing their local grocery store, according to world-renowned academic John Hattie.

Professor Hattie said ‘food choice’ had led to a “clogging of the motorways” as shoppers avoided their neighbourhood grocery store in pursuit of alternatives.

Food choice has also fuelled unhealthy competition between grocery stores, he told a packed lecture theatre at the University of Melbourne on Tuesday evening.

“Nearly all this choice is based on hearsay, the flavor of the food, and rarely on whether the food is or isn’t adding nutritional value to the shoppers’ health,” he said.

All right, all right. If you’re a regular reader of the JPG blog, you’ve probably already deduced that I changed the words “school” and “students” to “grocery stores,” “food,” and “shoppers.” Here’s what Professor Hattie actually argued:

Almost 60 per cent of Melbourne students are bypassing their local school, according to world-renowned academic John Hattie.

Professor Hattie said school choice had led to a “clogging of the motorways” as students avoided their neighbourhood school in pursuit of alternatives.

School choice has also fuelled unhealthy competition between schools, he told a packed lecture theatre at the University of Melbourne on Tuesday evening.

“Nearly all this choice is based on hearsay, the nature of the students, and rarely on whether the school is or isn’t adding value to the students’ learning,” he said.

Frankly, it’s no less absurd than the modified version.

So-called “experts” demonstrate their own hubris and contempt when they proclaim from their ivory towers that they know better than the little folks below, despite lacking any local knowledge about a particular student’s aptitude, interests, or learning needs. Aside from the fact that the best international evidence suggests that school choice boosts student outcomes (i.e., on average, the parents outperform the central planners), educational choice is an end in and of itself.

Moreover, although I don’t think education policy should be used as a lever to promote environmental goals, research from Professor Bart Danielson suggests that school choice policies are more environmentally friendly than district-based systems. Countless parents have fled urban areas to rescue their children from failing district schools–the “white flight” that Hattie decries–which contributes greatly to suburban sprawl. According to Danielson’s research, parents are more likely to remain in urban areas when they have more educational options, thereby reducing their carbon footprint.

In short: the “clogged roads” argument against choice is as spurious as it is silly.

I’ve written several times recently about how short term gains in test scores are not associated with improved later life outcomes for students.  Schools and programs that increase test score quite often do not yield higher high school graduation or college attendance rates.  Conversely, schools and programs that fail to produce greater gains in test scores sometimes produce impressive improvements in high school graduation and college attendance rates, college completion rates, and even higher employment and earnings.  I’ve described at least 8 studies that show a disconnect between raising test scores and stronger later life outcomes.

Well, now we have a 9th.  Earlier this month MDRC quietly released a long-term randomized experiment of the effects of the SEED boarding charter school in Washington, DC.  Because SEED is a boarding school, there was a lot of hope among reformers that it might be able to make a more profound difference for very disadvantaged students by having significantly more time to influence students and structure their lives.  Of course, boarding schools also cost significantly more — in this case roughly twice as much as traditional non-residential schools.

While the initial test score results are very encouraging, the later life outcomes are disappointing.  After two years students admitted to SEED by lottery outperformed those denied admission by lottery by 33% of a standard deviation in math and 23% in reading.  If we judged the quality of schools entirely based on short term changes in test scores, as many reformers would like to do, we’d say this school was doing a great job.

In fact, SEED may be doing a great job in a variety of ways, but when we look at longer term outcomes for students on a variety of measures the evidence demonstrating SEED’s success disappears or even turns negative.  Of the students accepted by lottery to SEED 69.3% graduate from high school after four years compared to 74.1% for the control group, a difference that is not statistically significant.  And when asked about their likelihood of attending college, there was no significant difference between the two groups.  SEED students also score significantly higher on a measure of engaging in risky behavior and lower on the grit scale.

We’ve seen this pattern before.  Research by Marty West and colleagues of no excuses charter schools in Boston found large gains in test scores but also significantly lowered student performance on noncognitive measures.  And Josh Angrist and colleagues found that those schools actually decrease four year high school graduation rates despite large gains in test scores.  In their words:

Perhaps surprisingly given the gains in test score graduation requirements reported in column 2 of table 4, the estimates in column 4 of this table suggest not. In fact, charter attendance reduces the likelihood a student graduates on time by 14.5 percentage points, a statistically significant effect.

It’s time that people start paying a lot more attention to this pattern of a disconnect between short term test score gains and long term life outcomes.  We can’t just dismiss this pattern as fluke.  And the reduction in noncognitive skills may be important for explaining this pattern.  Reduced grit scores may not just be the product of reference group bias.  It appears that certain types of charter schools that are able to produce large test score gains also lower character skills and fail to yield long term improvements in life outcomes.  Conversely other types of charter and private schools in choice programs fail to improve test scores but yield large gains in later life outcomes.

If we think we can know which schools of choice are good and ought to be expanded and which are bad and ought to be closed based primarily on annual test score gains, we are sadly mistaken.  Various portfolio management and “accountability” regimes depend almost entirely on this false belief that test scores reveal which are the good and bad schools.  The evidence is growing quite strong that these strategies cannot properly distinguish good from bad schools and may be inflicting great harm on students.  Given the disconnect between test scores and later life outcomes we need significantly greater humility about knowing which schools are succeeding.

Russ Whitehurst, one of the most cool-headed education researchers, throws cold water on the grit craze seizing some quarters of the ed reform world.  As I warned in my recent review of the Angela Duckworth and Paul Tough books, “This new attention to character skills has many of the markings of previous failed fads…. In short, school and educator practice with respect to character skills is running far ahead of knowledge.” I wanted to cool grit fever, but in his recent piece Russ throws icy cold water on it.  He raises excellent points but I wonder if Russ is too cold on the importance of character skills.

Russ makes three arguments: 1) A recent study that compared grit scores among fraternal and identical twins suggests that grit may be heritable to a large degree, which would make it unrealistic to expect schools or others to be able to alter it; 2) The twin study as well as a meta-analysis of grit research found that grit only explains about 2-3% of the variance in achievement scores, which Russ thinks makes it a poor predictor of other outcomes; and 3) The meta-analysis suggests that grit may be highly correlated with conscientiousness, one of the Big 5 personality traits that psychologists have been studying for a long time.

I think Russ is most persuasive on the last point.  Grit may be more of an effective marketing brand than a new contribution to the field.  But whether it is really distinct from conscientiousness or not, this does not establish whether grit and other character skills are important for education reform.

Russ is much less persuasive with his second argument.  The fact that grit or other character skills may not be strongly predictive of achievement test results is not surprising if these non-cog measures capture something that is important independently of cognitive ability.  That is, the true test of the predictive power of “noncog” measures is not whether they are correlated with cognitive measures (like achievement scores), but whether they are correlated with later life outcomes.  As it turns out out, they are.  As this recent piece in Economics of Education Review by Collin Hitt, Julie Trivitt, and Albert Cheng shows, their character skill measure collected in middle or early high school is predictive of later educational attainment, employment, and earnings in 5 different longitudinal panel data sets, even after cognitive ability and other factors are controlled.

Russ’ first argument is the most important for educational reform.  If grit or other character skills are not malleable, then why bother devoting a lot of energy to trying to address them in schools?  Russ is correct to point out that about 37% of the perseverance component of grit is heritable, but that does not establish that educational policy and practice are unable to alter the non-heritable factors that form grit and other character skills.

There is a growing body of evidence that suggests character skills are malleable and that education plays an important role in shaping and altering character.  Albert Cheng and Gema Zamarro have a new study that shows students randomly assigned to teachers who possess stronger character skills experience an increase in their own character skills.  And Albert Cheng has another study using a student fixed-effect research design that has the same finding.  This would not be possible if character skills were not malleable.

In addition we have an entire literature on pre-school and school choice suggesting that educational interventions can produce long-term success without improving short term achievement test scores (and vice versa).  It’s not well-understood exactly how these benefits are being produced, but a reasonable explanation is that early childhood and effective school choice programs may improve character skills without also increasing test scores.  And in the long-run the improvement in character skills may be more important for success.

Russ Whitehurst is right to warn us about the irrational exuberance some have about grit, but he shouldn’t throw out the baby with ice cold bath water.  There seems to be something very important about character skills in education even if we do not fully understand how to define, measure, or alter them.

(edited for typos)