Jay P. Greene's Blog

Under Tom Vander Ark‘s leadership the Gates Foundation pursued an education reform strategy focused on creating smaller high schools.  The theory was that smaller high schools would create tighter social bonds between schools and students, preventing students from slipping through the cracks and increasing the likelihood that they would graduate and go on to college.  Smaller high schools could also be more varied in their approaches and offerings, allowing students to choose schools that best fit their needs.

But around the same time Vander Ark left the Gates Foundation at the end of 2006, the reform strategy shifted.  Rather than fostering small, diverse schools of choice, the Gates Foundation now wanted to build centralized systems of what everyone should be taught (Common Core) and centralized systems of evaluating, training, and promoting teachers (Measuring Effective Teachers).  As I’ve written before, the shift in Gates strategy was not prompted by research.  In fact, the high quality random-assignment study that Gates had commissioned to evaluate the small high school strategy showed strong, positive results.  But the post-Vander Ark leadership at Gates couldn’t wait for the evidence.  The knew the truth without any pesky research and had abandoned the small high schools strategy in favor of their new centralization approach years before those results were released.

Well, the evidence continues to pile up that the Gates strategy under Tom Vander Ark was effective and the new strategy is a failure.  A new National Bureau of Economic Research study by Lisa Barrow, Amy Claessens, and Diane Whitmore Schanzenbach examines the effects of small schools in Chicago.  Just like the earlier random-assignment study of small high schools in New York City had found and like the Vander Ark theory of reform had suggested, small schools promote high school graduation:

We find that small schools students are substantially more likely to persist in school and eventually graduate. Nonetheless, there is no positive impact on student achievement as measured by test scores. The finding of no test score improvement but a strong improvement in school attainment is consistent with a growing literature suggesting that interventions aimed at older children are more effective at improving their non-cognitive skills than their cognitive skills.

As Vander Ark had expected, smaller schools have non-achievement effects, like creating stronger social bonds, that help students go further in their schooling.  And as numerous studies have shown, higher educational attainment is strongly predictive of a host of good outcomes for students later in their lives. Score: Tom Vander Ark 2, New Gates PLDD Strategy 0.

Meanwhile I witnessed further confirmation of the failure of the new Gates approach during a panel at the Association for Education Finance and Policy conference last weekend.  The Gates folks spent more than an hour presenting their Measuring Effective Teachers work.  That stuff may win over gullible policymakers and journalists, but the researchers at AEFP were not impressed.

Tim Sass had the first question and essentially repeated my concerns about how MET fails to provide evidence for the use of multiple measures.  Value-added test scores are predictive of later life earnings, as Chetty, et al have shown, he said, but why should we believe that classroom observations measure anything we care about?  The Gates folks didn’t really have an answer.  Jane Hannaway then articulated my concern that effective teaching may be too context-dependent to lend itself to a single formula for effective practices.  The Gates folks responded that there are probably some basic skills for effective teaching that are useful in all contexts.  They may have a point but that does not address whether MET is getting at those basic skills or not.  The weak correlations of everything in the study suggest they are not finding approaches that are commonly effective.

A third questioner wondered about the cost of adopting the MET approach, especially given the need for multiple observations by multiple, trained raters.  The response was that schools are already spending money on classroom observations but of course that does not address the extra costs of the multiple rater/observations approach.

And lastly William Mathis asked about the generally weak correlations between classroom observations and value-added test scores.  Doesn’t this suggest that these are distinct dimensions of effective teaching that shouldn’t be combined in a single measure, he wondered.  The Gates response surprised me.  They said that the weak correlations were good news.  It is precisely because classroom observations and value-added measures capture different dimensions of effective teaching that we need to combine them in an overall measure.

Of course, this ignores the Sass question about whether we know that the classroom observations are measuring any important aspect of effective teaching.  But more problematic was the “heads I win, tails you lose” nature of their response.  If earlier arguments defending MET were based on how classroom observations and VAM were correlated, then how could the lack of correlation also be presented as proof of MET’s success?  I went up to the Gates presenters after the panel (and chatted with some of them later) and asked them what MET could have found that would have led them to conclude that combining the measures was a bad idea.  They were stumped.  Maybe negative correlations would have dissuaded them from advocating a combined measure, but they weren’t confident about that either.

Essentially, they admitted that the MET policy recommendation is a non-falsifiable claim.  No research finding would have dissuaded them from it.  The ability to falsify a claim is at the heart of science.  MET is not science; it is just politics.  You should have seen the discomfort of the Gates researchers as they pondered why they were presenting non-falsifiable claims as research findings at an academic conference.  This sort of thing corrupts science and has reputational consequences for the researchers who lend their credibility to the new evidence-free Gates reform strategy.

There is a solution to the rot at Gates — Bring back Tom Vander Ark.  At least his ideas were supported by rigorous research.