Jay P. Greene's Blog

In my last post I wrote about the pattern at the Gates Foundation of abusing the idea of “research” and “evidence” to advance its education policy agenda.  Gates has an organizational culture that permits intellectual corruption.  There are good people at Gates doing good work, but there is something rotten about the organization that needs to be changed if they hope to succeed over the long run.

In addition to their abuse of research and evidence, the Gates Foundation suffers from a bloated staff and paralyzing bureaucracy.  As their 990 tax filings show, their assets doubled over the last decade, but their staffing levels increased ten-fold — even more rapidly than the increase in assets as Buffett adds his money to Gates to create a philanthropic Leviathan.  They have so many people that they needed to build the $500 million palace pictured above to hold all of them.

But with huge size, staffing, and wealth comes the huge danger of corruption.  If an organization becomes bloated, inefficient and corrupt in the profit-seeking sector, the possibility of a hostile takeover can help check or eliminate abuses.  But in the non-profit sector there are no corporate raiders.  No outside shareholders can come in to take over the Gates Foundation, sell off its over-priced facilitates, cut staffing, reduce corruption and focus on the core mission.

Instead, non-profits need to check the danger of corruption that comes with wealth and power in the same way that governments do — by creating institutional constraints, dividing power, and pitting ambition against ambition.  In short, non-profits need a Constitution.

Specifically, the Gates Foundation has just become too damn big for its own good.  It’s so big and powerful that just about everyone in the education policy world gets money from them or hopes to.  It’s so big that everyone within the organization is too eager to gain control over it, causing in-fighting and the need for rigid top-down controls.  It’s so big that they can indulge foolish ideas and make irresponsible claims without fear of consequences.

One way Gates could check these problems is to divide its education unit into a Team A and Team B, each of which would operate independently with its own theory of action and reform agenda.  The different Teams within Gates could then compete with each other to develop and pursue the best reform ideas.  They could also help keep each other honest by having an interest to detect, reveal, and stop any intellectual dishonesty in the other.

Many people wrongly believe that organizations function best when they achieve greater scale and are streamlined, but this is incorrect in the peculiar world of government and non-profit organizations.  As Patrick Wolf and James Q. Wilson’s work on bureaucracy shows, redundancy within government can be a desirable arrangement.  Having the FBI, DEA, and ATF all chasing drug dealers is beneficial because they compete with each to do the best job and win a larger share of congressional appropriations.  Redundancy can simulate the choice and competition of the private market.

Similarly the division of power and responsibility between local, state, and federal governments as well as between the different branches of each government helps check abuse and corruption while providing some of the positive effects of choice and competition.  Smart non-profits should likewise develop a policy to split themselves into smaller competing units once they reach a certain size.

Another institutional arrangement that might help right the ship at Gates is to develop an independent internal research department that reports directly to the board and not to the heads of any of its programmatic units.  A research department that does not report to the programmatic unit heads is less likely to feel compelled to verify the wisdom of the paths chosen by the programmatic units.

In addition, non-profits like Gates should develop a policy that prevents them from ever conducting public evaluations of their own projects internally.  The research unit’s responsibilities should be limited to contracting out research to independent third parties and then reporting the results to the board.  One of the real problems with the MET project at Gates is that is was funded and conducted internally by Gates.  That made it very hard for them to report that the project had failed to find what they had hoped.  You shouldn’t be the judge and jury in your own case.

Lastly, changing the organizational culture to one that gives primacy to intellectual integrity requires cleaning house at the leadership level.  Vicki Phillips, the head of the education unit, has to go.  She has repeatedly mis-described the findings of their own research.  I’m not sure whether it is because she does not understand the research or because she doesn’t care about being accurate (and I’m not sure which would be worse), but you can’t effectively lead an organization if you can’t honestly describe your own research.  It might be good for Gates to consider appointing a well-respected scholar to head its education units, just as the Carnegie Foundation did when it selected University of Chicago researcher, Anthony Bryk, as its president.

Tom Kane, who until recently served under Vicki Phillips, brought impressive research credentials to the table but unfortunately has chosen to compromise those credentials.  Kane is an incredibly capable and accomplished researcher, but even the best can be tarnished.  Gates needs to cut all remaining ties with Kane to set the example that accurate and honest reporting of research is of primary value.

Of course, none of this can or will happen unless Bill Gates wants it to.  Perhaps Gates himself is the source of the problem.  If that’s true, then no organizational or staffing change will improve the situation.  But I suspect that Gates does not want to see his wealth squandered.  He doesn’t want to be the next Walter Annenberg, whose $500 million “Challenge” ultimately “had little impact on school improvement and student outcomes…”

The MET project is already almost $400 million that Gates has spent with little to show for it.  I don’t think Gates wants to keep doing that.  And even if his internal research declares success and others are too timid to publicly question that claim, over the long run (especially when the money stops flowing in this direction) people will think of Gates as they think of Annenberg — as someone who failed to use his enormous wealth for positive effect in education.

In the end, this is all up to Bill Gates.  He can choose to make a giant bonfire of his fortune by squandering it on palatial buildings, excessive staffing, and foolish enterprises whose failure is only temporarily disguised by dishonesty.  Or he can choose to make the organizational and staffing changes necessary to get the Gates Foundation back on track.

(edited for some typos)

In education reform debates it is far too common to hear someone say “the evidence shows” something that is just their preferred policy that is not supported by research at all.  People confuse what makes good sense and is good politics with what is actually supported by evidence.  At the Gates Foundation this problem is endemic.  They have repeatedly confused evidence and politics.

I think I can clearly illustrate this confusion of evidence and policy preference at the Gates Foundation in the most recent article by Tom Kane in Education Next summarizing the Measuring Effective Teachers (MET) project results.   MET is an ambitious project to record several thousand classroom lessons, survey students, and administer multiple standardized tests to identify the best way to measure teacher effectiveness and eventually identify teaching practices that are associated with greater learning.  The study costs $45 million on top of the $335 million reported cost of implementing the program in several school districts.

The main claimed finding of MET at this point is that combining classroom observation and student survey scores with student achievement gains is the best way to measure teacher effectiveness.  As Kane writes, “the evidence reveals that…  rather than rely on any single indicator, schools should try to see effective teaching from multiple angles.”  I’m willing to agree with Kane that using multiple measures of teacher effectiveness is supported by political wisdom and sound theory, but the evidence they produced does not demonstrate the merits of multiple measures.

Kanes summarizes “the case for multiple measures” in the second to last section of his article.  He states, “First, combining [multiple measures] generates less volatility from course section to section or year to year, and greater predictive power.” The results don’t exactly support this claim.  As can be seen below in Figure 1 reproduced from his article and in Table 16 on p. 51 of the Measuring Effective Teachers report, an equally weighted combination of student achievement gains with classroom observation scores and student survey results actually lowers predictive power.  You are better at predicting teacher value added in a class just by using the teacher value added measure from another class than by combining that achievement gain measure equally with classroom observation scores and student survey responses, which is the opposite of “combining them generates… greater predictive power.”

The only way using multiple measures could have a roughly equal predictive power to achievement gains alone is if they are combined such that achievement gains constitute 75.8% of the combined measure, with only 4.2% of the combined measure coming from classroom observation scores and 20.0% coming from student surveys.

But there are significant difficulties and costs associated with collecting the classroom observation scores.  Every observer had to receive 17 to 25 hours of training and even after that 23% of the observers had to be excluded for lack of reliability.  And then as Kane acknowledges: “Even with trained raters, we had to score four lessons, each by a different observer, and average those scores to get a reliable measure of a teacher’s practice. Given the high opportunity cost of a principal’s time, or the salaries of professional peer observers, classroom observations are the costliest source of feedback.”  All of this was necessary for a measure that constituted 4.2% of a combined measure with about the same predictive power as forgetting about classroom observations and just using achievement gains.

Someone reviewing this evidence who was not already committed to the policy of using multiple measures would obviously conclude that classroom observations were not worth the significant expense and bother.  The conclusion Kane and Gates offer is not driven by “the evidence” but by their preference for a policy that is based on other political and theoretical reasons.

I should note that the increase in reliability from combining measures of teacher effectiveness provides little consolation.  Kane measures reliability as the correlation of the evaluation score from class to class for the same teacher.  You could improve reliability without improving or even while hurting predictive power simply by adding another  variable to the combined measure.  It would be more consistent (reliable), but it would be more consistently wrong.

Kane then offers another argument for combining measures: “A second reason to combine the measures is to reduce the risk of unintended consequences, to lessen the likelihood of manipulation or ‘gaming.’ Whenever one places all the stakes on any single measure, the risk of distortion and abuse goes up.”  This makes very good sense and is a persuasive argument.  The only problem is that it is not in any way derived from “the evidence” produced by their study.  It’s just a sound theoretical argument.  Kane and Gates shouldn’t say “the evidence” supports multiple measures when they aren’t actually relying on evidence to make their claim.  They didn’t need to spend almost $400 million to implement and study MET to make this point.

And finally, Kane suggests that “[t]here is a third reason to collect multiple measures: conflicting messages from the multiple sources of information send a signal to supervisors that they should take a close look at what’s going on in the classroom.”  Again, this is a theoretical argument rather than from any evidence the study collected.  And unfortunately, the evidence from the study suggests that a combined message will send “conflicting messages” almost all of the time.  The correlation between classroom observation or student survey scores and achievement gains was no higher than .13.  With such a low correlation, administrators will very often see differences between teacher effectiveness as measured by each of the three types of measures.  Kane might as well suggest that supervisors should always take a close look at every teacher.

I’m inclined to agree with Kane and Gates that it is better to use multiple measures when evaluating teacher effectiveness.  I just don’t see how “the evidence” does anything to support this view.  The argument for multiple measures is largely theoretical and political.  Theory suggests that a single measure is more subject to manipulation and unwanted distortion in teacher behavior.  And politics suggests that teachers will be more resistant to any system that is based solely on test scores.  These are all fine reasons for supporting multiple measures, we just shouldn’t debase the currency of research by falsely claiming that they are supported by the evidence when the evidence shows no such thing.  They are just confusing evidence and policy preferences justified by considerations that have nothing to do with research. The truth is that MET was a very expensive effort that failed to produce the evidence they wanted, but in Orwellian fashion they declare victory and still say it supports their pre-determined conclusion.

Unfortunately, there is a pattern at  Gates of this abuse of “evidence” and “research” to support preferred policies.  I’ve written a number of posts in the past detailing this problem at Gates.  In addition to spinning the multiple measures claim, I’ve pointed out that they falsely claimed that their student survey results showed that “[t]eaching to the test makes your students do worse on the tests.”  I noted that Gates was again indifferent to evidence when they abandoned their small schools strategy without waiting for the results of a random-assignment evaluation that ultimately showed that small schools were effective.  And Gates has backed the push for Common Core standards with phony science.

Unlike other critics of the Gates Foundation, I am not motivated by the belief that it is illegitimate for billionaires to use their wealth to try to advance education reform.  On the contrary, I’ve focused on Gates because I believe they are squandering their great potential to have a positive impact.  I’d like them to do better.

But even more importantly, I’ve harped on these abuses of “evidence” and “research” to advance the Gates policy agenda because I fear that Gates is undermining the use of real evidence and research by others to positively influence policy.  I understand that people and organizations can favor policies without having the evidence to prove their merit.  But I cannot understand or accept abusing the idea of evidence and research to advance preferred policies.  Doing so ruins the use of evidence by everyone by feeding the cynical belief that all research is just a way to manipulate others to get what you want.  The more that the general credibility of research and evidence are damaged, the more that policy outcomes will be determined by the brute power of involved interests, which means that the unions are more likely to prevail.  A belief in research and evidence is the only way for weaker interests to triumph, so it is essential that the ed reform movement not debase their own currency.

I still have hope that Gates can right their ship.  It won’t be easy, but they can take important steps to change their organizational culture and structure so that they do not repeatedly abuse claims of research and evidence for their policy preferences.  In the next post, I’ll explain what they should do to reform themselves.

(Edited to correct typo)