Jay P. Greene's Blog

Last week Bill Gates gave a speech to articulate his foundation’s strategy for education reform and the basis for that strategy. Unfortunately, throughout the speech Gates misreads and distorts findings from research, including research that the foundation has funded or conducted itself.  Why hasn’t anyone called Gates on these errors?

Look, I’m not surprised that a non-researcher misunderstands research.  And I’m not surprised that a foundation is pursuing a strategy that is not well-supported by research.  Frankly, he’s entitled to pursue any strategy he wants for any reason he prefers.  But Gates gets up there, puts on his metaphorical lab coat, and declares that Science says… It’s then the job of actual scientists, even social scientists, to correct him.  If we don’t, then we are letting Gates corrupt science.  The Gates foundation doesn’t, for the most part, buy false research findings.  But Gates is contributing to the corruption of social science as researchers obviously feel the need to stay quiet when he and other foundation leaders misread and distort findings.

How do I know Gates is distorting and misreading research?  For one thing, several researchers involved in the studies he is referencing have told me so.  They know that he, Vicki Phillips, and other senior Gates officials are mis-describing the results of their work.  It’s no secret among researchers that this is occurring, they are just reluctant to do anything about it.

When I asked one researcher involved in the Measuring Effective Teachers (MET) project why they hadn’t corrected Vicki Phillips’ inaccurate description of their work, he agreed “it was Vicki’s mistake” but “anyone who reads the original paper can see the actual findings. I guess this is like lots of policy-relevant papers…”  That is, he just accepted the corruption and trusted that his results would speak for themselves.  Others with whom I have spoken who have worked on MET and other Gates-funded projects have not been quite so passive, but none feel brave enough to correct the Emperor.

I also know that Gates is distorting and misreading research because what he says clearly contradicts what researchers have found.  For example let’s look at how Gates claims that research supports his foundation’s shift in strategy from small schools of choice to measuring effective teaching:

Early on, we thought smaller schools were the way to drive up college-ready rates. We set out to build the model of a successful school by breaking large high schools into new, smaller ones. Those efforts did raise graduation rates.  But only some of the smaller schools also raised college-readiness rates—and the ones that did put a huge focus on training skilled teachers. So we weren’t going to reach our goals simply by changing the size of the school.

Gates is right that small schools of choice in New York City caused increased high school graduation, college-readiness, and college-attendance.  We know this from a rigorous randomized experiment conducted by MDRC, although it is important to note that these results came out several years after Gates had already abandoned the small schools strategy in 2008.  Gates starts deviating from an accurate reading of the research when he says “only some of the smaller schools also raised college-readiness rates.”  The research says nothing about how only some schools were effective and it certainly never claimed that focusing on “training skilled teachers” accounts for why some might have been more effective than others.

To justify the shift in strategy, Gates appears to be relying on the results of a set of focus group interviews MDRC and its partners conducted with teachers and principals.  These results were released in 2013, so unless the Gates Foundation has a time machine hidden within its headquarters, even this research could not account for the 2008 change in strategy.  Rather than basing the strategy on research as Gates’ speech is designed to suggest, this could at best serve as a post hoc justification.

But even the focus group interviews do not support what Gates claims.  Focus groups were held with educators at 25 of the 30 small schools of choice (SSCs) that MDRC judged to be most effective at raising high school graduation rates, not college-readiness rates (p. 63).  They didn’t compare those most effective schools to less effective ones to see if the presence or absence of any reported characteristic was associated with their effectiveness.  So, this is basically a “best practices” or selection on dependent variable analysis that cannot be used to make causal claims.  The MDRC report explicitly warns readers about this:

A second important caveat is that information from SSC interviews and focus groups only represents respondents’ opinions about the factors that make SSCs effective. This information can only suggest hypotheses; it cannot confirm them. (p. 70)

The report also never mentions “focusing on training skilled teachers” as an explanation offered by educators at the 25 most-effective small schools to explain their effectiveness.  Instead, what it does say is that 84% of principals and 64% of teachers believed that “teachers” were a factor in their school’s success (p. 19).  Teachers were the first factor mentioned most often by principals (52%) but only 8% of teachers mentioned that first.  They gave stronger emphasis to personal relationships (52%), high academic expectations (12%), and leadership (12%).  So, these focus group interviews didn’t mention “training skilled teachers” and did not even clearly give emphasis to “teachers.”

Let’s review.  Gates justifies a 2008 switch in strategy based on a 2013 study.  The study consists of focus group interviews, not a rigorous research design, and only involved the most effective schools, so as in any selection on dependent variable analysis, no causal claims can be made.  And the report doesn’t mention “focusing on training skilled teachers” and doesn’t even clearly suggest teachers as the main reason educators offered for the success of their schools.  Besides that, Mrs. Lincoln, how was the play?

In the speech Gates goes on to assert that Kentucky, Denver, Green Dot schools in LA, and DC all have achieved impressive gains because they have focused on improving teacher quality.  Even if these school systems did improve outcomes and had focused on teacher quality, this would be another example of selection on dependent variable analysis that cannot be used to support causal claims.  Maybe these school systems did other things that account for their success.  And Gates never considers that many other school systems also paid attention to teacher training without improvement in results.  This is not “evidence” to support a shift in the foundation’s strategy to measuring effective teachers.

And then Gates claims that using “multiple measures of effectiveness” when evaluating teachers is “backed by evidence” and has been essential in improving teacher skills.  I’ve dealt with this in several previous posts, but probably most directly here.  It may well be sensible to use multiple measures to assess teacher effectiveness, but this is not “backed by research” and there is no evidence that using multiple measures improves student outcomes.

I’d continue documenting how Gates misreads and distorts research findings, but by now the pattern should be clear to any observant reader.  The question is what is anyone going to do about this?  Will more researchers come forward to question the accuracy of how the Gates Foundation references research?  Will reporters finally start to ask about this?

This entry was posted on Wednesday, October 14th, 2015 at 11:23 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.