Emperor Gates Has No Clothes

Last week Bill Gates gave a speech to articulate his foundation’s strategy for education reform and the basis for that strategy. Unfortunately, throughout the speech Gates misreads and distorts findings from research, including research that the foundation has funded or conducted itself.  Why hasn’t anyone called Gates on these errors?

Look, I’m not surprised that a non-researcher misunderstands research.  And I’m not surprised that a foundation is pursuing a strategy that is not well-supported by research.  Frankly, he’s entitled to pursue any strategy he wants for any reason he prefers.  But Gates gets up there, puts on his metaphorical lab coat, and declares that Science says… It’s then the job of actual scientists, even social scientists, to correct him.  If we don’t, then we are letting Gates corrupt science.  The Gates foundation doesn’t, for the most part, buy false research findings.  But Gates is contributing to the corruption of social science as researchers obviously feel the need to stay quiet when he and other foundation leaders misread and distort findings.

How do I know Gates is distorting and misreading research?  For one thing, several researchers involved in the studies he is referencing have told me so.  They know that he, Vicki Phillips, and other senior Gates officials are mis-describing the results of their work.  It’s no secret among researchers that this is occurring, they are just reluctant to do anything about it.

When I asked one researcher involved in the Measuring Effective Teachers (MET) project why they hadn’t corrected Vicki Phillips’ inaccurate description of their work, he agreed “it was Vicki’s mistake” but “anyone who reads the original paper can see the actual findings. I guess this is like lots of policy-relevant papers…”  That is, he just accepted the corruption and trusted that his results would speak for themselves.  Others with whom I have spoken who have worked on MET and other Gates-funded projects have not been quite so passive, but none feel brave enough to correct the Emperor.

I also know that Gates is distorting and misreading research because what he says clearly contradicts what researchers have found.  For example let’s look at how Gates claims that research supports his foundation’s shift in strategy from small schools of choice to measuring effective teaching:

Early on, we thought smaller schools were the way to drive up college-ready rates. We set out to build the model of a successful school by breaking large high schools into new, smaller ones. Those efforts did raise graduation rates.  But only some of the smaller schools also raised college-readiness rates—and the ones that did put a huge focus on training skilled teachers. So we weren’t going to reach our goals simply by changing the size of the school.

Gates is right that small schools of choice in New York City caused increased high school graduation, college-readiness, and college-attendance.  We know this from a rigorous randomized experiment conducted by MDRC, although it is important to note that these results came out several years after Gates had already abandoned the small schools strategy in 2008.  Gates starts deviating from an accurate reading of the research when he says “only some of the smaller schools also raised college-readiness rates.”  The research says nothing about how only some schools were effective and it certainly never claimed that focusing on “training skilled teachers” accounts for why some might have been more effective than others.

To justify the shift in strategy, Gates appears to be relying on the results of a set of focus group interviews MDRC and its partners conducted with teachers and principals.  These results were released in 2013, so unless the Gates Foundation has a time machine hidden within its headquarters, even this research could not account for the 2008 change in strategy.  Rather than basing the strategy on research as Gates’ speech is designed to suggest, this could at best serve as a post hoc justification.

But even the focus group interviews do not support what Gates claims.  Focus groups were held with educators at 25 of the 30 small schools of choice (SSCs) that MDRC judged to be most effective at raising high school graduation rates, not college-readiness rates (p. 63).  They didn’t compare those most effective schools to less effective ones to see if the presence or absence of any reported characteristic was associated with their effectiveness.  So, this is basically a “best practices” or selection on dependent variable analysis that cannot be used to make causal claims.  The MDRC report explicitly warns readers about this:

A second important caveat is that information from SSC interviews and focus groups only represents respondents’ opinions about the factors that make SSCs effective. This information can only suggest hypotheses; it cannot confirm them. (p. 70)

The report also never mentions “focusing on training skilled teachers” as an explanation offered by educators at the 25 most-effective small schools to explain their effectiveness.  Instead, what it does say is that 84% of principals and 64% of teachers believed that “teachers” were a factor in their school’s success (p. 19).  Teachers were the first factor mentioned most often by principals (52%) but only 8% of teachers mentioned that first.  They gave stronger emphasis to personal relationships (52%), high academic expectations (12%), and leadership (12%).  So, these focus group interviews didn’t mention “training skilled teachers” and did not even clearly give emphasis to “teachers.”

Let’s review.  Gates justifies a 2008 switch in strategy based on a 2013 study.  The study consists of focus group interviews, not a rigorous research design, and only involved the most effective schools, so as in any selection on dependent variable analysis, no causal claims can be made.  And the report doesn’t mention “focusing on training skilled teachers” and doesn’t even clearly suggest teachers as the main reason educators offered for the success of their schools.  Besides that, Mrs. Lincoln, how was the play?

In the speech Gates goes on to assert that Kentucky, Denver, Green Dot schools in LA, and DC all have achieved impressive gains because they have focused on improving teacher quality.  Even if these school systems did improve outcomes and had focused on teacher quality, this would be another example of selection on dependent variable analysis that cannot be used to support causal claims.  Maybe these school systems did other things that account for their success.  And Gates never considers that many other school systems also paid attention to teacher training without improvement in results.  This is not “evidence” to support a shift in the foundation’s strategy to measuring effective teachers.

And then Gates claims that using “multiple measures of effectiveness” when evaluating teachers is “backed by evidence” and has been essential in improving teacher skills.  I’ve dealt with this in several previous posts, but probably most directly here.  It may well be sensible to use multiple measures to assess teacher effectiveness, but this is not “backed by research” and there is no evidence that using multiple measures improves student outcomes.

I’d continue documenting how Gates misreads and distorts research findings, but by now the pattern should be clear to any observant reader.  The question is what is anyone going to do about this?  Will more researchers come forward to question the accuracy of how the Gates Foundation references research?  Will reporters finally start to ask about this?

13 Responses to Emperor Gates Has No Clothes

  1. George Mitchell says:

    What is anyone going to do about it?

    Surely that is a rhetorical question. Jay is one of the few who will call out this issue.

    The proliferation of fawning and servile “researchers” and “distinguished scholars” who cower before sources of K-12 philanthropy is quite regrettable.

  2. George Mitchell says:

    I see Tom Loveless has weighed in with a tweet about Jay’s post. Good for him. Some folks are quietly squirming.

  3. George Mitchell says:

    It’s not just philanthropists. Many researchers are craven with the media. A prominent scholar once agreed with me that a central claim in an article was false; the same individual praised the reporter for a “great” piece.

  4. Terry Stoops says:

    Geez, Jay, what do you expect from someone who didn’t even graduate from college?

    (Yup, that’s a high quality joke right there.)

  5. Greg Forster says:

    What the dickens do you mean, “even social scientists”?!?!

  6. Tunya Audain says:

    Call-out Of Gates Urgently Needed

    Look at the title of this story — Imposing Common Core On America Is Way Harder Than CURING MALARIA – Bill Gates : http://dailycaller.com/2015/10/08/bill-gates-imposing-common-core-on-america-harder-than-curing-malaria/

    It is timely to ask: Who is advising Gates ? The story says Gates complained that “obstinate, noncompliant parents” were the problem behind resistance to Common Core. Yet his own children are in an expensive non-CC school. What advice caused him to avoid CC? Probably good advice. After 7 years of quietly funding experimental education programs he is now being more public about his well-funded agendas for others. Yes, for others, not his own family !

    He is doing the biological right thing by his children — protecting them from harm and hoping to launch them well-equipped into the future. Why would he then want to impose an amateur, not field-tested, poorly researched, one-size-fits-all federal program on other people’s children ?

    From someone outside the system — I am a parent and grandparent — I am ashamed at how “the system” lacks in judgment on this issue. Some sort of mass hysterical delusional intimidation must be going on. Is there really some organized conspiracy to produce a two class system — a small educated ruling elite and an obedient, compliant mass public ?

    There are few who are calling-out BOTH Gates and Common Core pushers on this questionable path.

    Only Jay P Greene, professor at University of Arkansas who supports education choice, seems to have the guts to call-out this hocus-pocus.

    As gullible public we need to arm ourselves against these totalitarian, wrong-headed moves. What are other hopeful signs besides JPGreene?

  7. […] There are few who are calling-out BOTH Gates and Common Core pushers on this questionable path. Only Jay P Greene, professor at University of Arkansas who supports education choice, seems to have the guts to call-out this hocus-pocus. Please see https://jaypgreene.com/2015/10/14/emperor-gates-has-no-clothes/ […]

  8. George Mitchell says:

    An early harbinger of the K-12 role Gates would play came when he/his foundation decided to not pursue vouchers because the issue was too contentious. Microsoft would not be a household name if it had treated consumers with similar disdain.

  9. […] could be more conservative on how he generalizes fairly narrow research findings, especially given how hard he is on others who misuse […]

  10. Sandra Stotsky says:

    Last night at an anti-Common Core forum in Westfield, MA, a distraught mother wanted to know why it is OK for Gates and Duncan to send their kids to private, non-CC schools, but she and her husband don’t make enough money to do so. How is that fair? Only the wealthy, it seems, will have school choice under CC. Sandra Stotsky


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: