In Education Myths I argued that we needed to rely on science rather than our direct experience to identify effective policies. Our eyes can mislead us, while scientific evidence has the systematic rigor to guide us more accurately.
That’s true, but I am now more aware of the opposite failing — believing that we can resolve all policy disputes and identify the “right way” to educate all children solely by relying on science. Science has its limits. Science cannot adjudicate among the competing values that might attract us to one educational approach over another. Science usually tells us about outcomes for the typical or average student and cannot easily tell us about what is most effective for individual students with diverse needs. Science is slow and uncertain, while policy and practice decisions have to be made right now whether a consensus of scientific evidence exists or not. We should rely on science when we can but we also need to be humble about what science can and can’t address.
I was thinking about this while reflecting on the Gates Foundation’s Measuring Effective Teachers Project. The project is an ambitious $45 million enterprise to improve the stability of value-added measures while identifying effective practices that contribute to higher value-added performance. These are worthy goals. The project intends to advance those goals by administering two standardized tests to students in 8 different school systems, surveying the students, and videotaping classroom lessons.
The idea is to see if combining information from the tests, survey, and classroom observations could produce more stable measures of teacher contributions to learning than is possible by just using the state test. And since they are observing classrooms and surveying students, they can also identify certain teacher practices and techniques that might be associated with greater improvement. The Gates folks are using science to improve the measures of student progress and to identify what makes a more effective teacher.
This is a great use of science, but there are limits to what we can expect. When identifying practices that are more effective, we have to remember that this is just more effective for the typical student. Different practices may be more effective for different students. In principle science could help address this also, but even this study, with 3,000 teachers, is not nearly large enough to produce a fine-grained analysis of what kind of approach is most effective for many different kinds of kids.
My fear is that the researchers, their foundation-backers, and most-importantly, the policymaker and educator consumers of the research are insensitive to these limitations of science. I fear that the project will identify the “right” way to teach and then it will be used to enforce that right way on everyone, even though it is highly likely that there are different “right” ways for different kids.
We already have a taste of this from the preliminary report that Gates issued last month. Following its release Vicki Phillips, the head of education at the Gates Foundation, told the New York Times: “Teaching to the test makes your students do worse on the tests.” Science had produced its answer — teachers should stop teaching to the test, stop drill and kill, and stop test prep (which the Gates officials and reporters used as interchangeable terms).
Unfortunately, Vicki Phillips mis-read her own Foundation’s report. On p. 34 the correlation between test prep and value-added is positive, not negative. If the study shows any relationship between test prep and student progress, it is that test prep contributes to higher value-added. Let’s leave aside the fact that these were simply a series of pairwise correlations and not the sort of multivariate analysis that you would expect if you were really trying to identify effective teaching practices. Vicki Phillips was just plain wrong in what she said. Even worse, despite having the error pointed out, neither the Gates Foundation nor the New York Times has considered it worthwhile to post a public correction. Science says what I say it says.
And this is the greatest danger of a lack of humility in the application of science to public policy. Science can be corrupted so that it simply becomes a shield disguising the policy preferences of those in authority. How many times have you heard a school official justify a particular policy by saying that it is supported by research when in fact no such research exists? This (mis)use of science is a way for authority figures to tell their critics, “shut up!”
But even if the Gates report had conducted multivariate analyses on effective teaching practices and even if Vicki Phillips could accurately describe the results of those analyses, the Gates project of using science to identify the “best” practices is doomed to failure. The very nature of education is that different techniques are more effective in different kinds of situations for different kinds of kids. Science can identify the best approach for the average student but it cannot identify the best approach for each individual student. And if students are highly varied in their needs, which I believe they are, this is a major limitation.
But as the Gates Foundation pushes national standards with new national tests, they seem inclined to impose the “best” practices that science identified on all students. The combination of Gates building a national infrastructure for driving educator behavior while launching a gigantic scientific effort to identify the best practices is worrisome.
There is nothing wrong with using science to inform local practice. But science needs markets to keep it honest. If competing educators can be informed by science, then they can pick among competing claims about what science tells us. And they can learn from their experience whether the practices that are recommended for the typical student by science work in the particular circumstances in which they are operating.
But if the science of best educator practice is combined with a national infrastructure of standards and testing, then local actors cannot adjudicate among competing claims about what science says. What the central authorities decide science says will be infused in the national standards and tests and all must adhere to that vision if they wish to excel along these centralized criteria. Even if the central authority completely misunderstands what science has to say, we will all have to accept that interpretation.
I don’t mean to be overly alarmist. Gates has a lot of sensible people working for them and there are many barriers remaining before we fully implement national standards and testing. My concern is that the Gates Foundation is being informed by an incorrect theory of reform. Reform does not come from science identifying the right thing to do and then a centralized authority imposing that right thing on everyone. Progress comes from decentralized decision-makers having the freedom and motivation to choose among competing claims about what is right according to science.
(edited for typos)
Jay P. Greene's Blog 
Who died and left Gates the GOD of education?
Honestly, he’s never been a teacher yet he has all of the good ideas?
This is ridiculous.
IF he wanted to improve education, he’d be working towards the destruction of the monopoly that’s dumbed down education for numerous students.
He should take that money and provide scholarships to children and put them in the many schools that are succeeding.
Look at the Catholic Schools, home-schooling, Christian Schools, etc.
Get the kids out of the public schools that seek NOT to educate but indoctrinate.
Perhaps the Gates people have been attending too many of the Grantmakers in Education programs where the older foundations’ funding priorities are how to move the Us beyond capitalism and nationalism.
I read the 2009 Carnegie report “The Opportunity Equation:Mobilizing for Excellence and Equity in Mathematics and Science Education” as I was researching the science standards under CCSS.
I could not understand why it assumed the US would no longer be a free market economy and we would be living in a “collaboration-oriented global marketplace”.
The priorities that get funding do not seem to be about raising the academic knowledge and skills of American students.
That might have an inequitable outcome.
For some the word “reform” has to do with the transformative effect the schools will have on American society once students and eventually adults have the desired attitudes and beliefs.
[…] U. of Ark. political scientist and education scholar Jay Greene has been blogging about the proper role of science in education policy, and his thoughts (continued here) are well worth reading. In particular, he warns that trying to […]
The “myth” is that the public school systems and Dept of Education should be incharge of their own quality control management and testing. Do you see colleges administering their own Admissions Testing? No.
If the success of schools were measured by OUTSIDE metrics and testing, the schools could not TEACH TO THE TEST….and the measurements would be (1) are the students EMPLOYABLE by industry as being competent to read, write, think independently, and communicare, and (2) are the kids able to pass college entrance exams that are NOT DUMBED DOWN by STATES trying to hide their poor school…then do they actually GET and KEEP jobs, and do they ACTUALLY graduate College?
Keep auditing success and providing the production of education SEPARATE like ALL of industry and business do….hence OUTSIDE auditors.
Is that so difficult?
How many times have you heard a school official justify a particular policy by saying that it is supported by research when in fact no such research exists?
We may have fought this battle to a draw in my district. At least 3 different administrators have claimed there is peer-reviewed research showing achievement gains in students whose classrooms are equipped with Smart Boards.
Needless to say, none of them could actually produce a study.
Another administrator claimed, in the middle of a packed and extremely contentious budget meeting, that instructional coaches produce enormous learning gains. Turned out she was citing Sanders’ research on learning gains produced by the best teachers. The administrator’s reasoning: since the best teachers produce major achievement gains, **any** instructional coach will produce excellent teachers who will in turn produce major achievement gains.
At this point we have a board member who is willing to ask the superintendent actually to produce a copy of the research she cites.
That seems to have put a damper on the breezy citations of research that doesn’t exist.
Excellent examples, Catherine. When choice and competition motivate providers to find effective approaches, science provides information. Without choice and competition, science is a club that authorities use to smack their opponents.
[…] why he has been changing his mind on issues like value-added modeling (which he used to support): https://jaypgreene.com/2011/01/18/the-dead-end-of-scientific-progressivism/ He also wrote a blog post on why he changed his mind on merit pay (caveat: he mentions there an […]
[…] U. of Ark. political scientist and education scholar Jay Greene has been blogging about the proper role of science in education policy, and his thoughts (continued here) are well worth reading. In particular, he warns that trying to […]
> Science can be corrupted so that it simply becomes a shield disguising the policy preferences of those in authority.
No, it can’t. You aren’t talking about science, you are talking about opinion masquerading as science.
> How many times have you heard a school official justify a particular policy by saying that it is supported by research when in fact no such research exists?
Science is a messy process. There likely are studies supporting opposing viewpoints. It often takes many years and many attempts to replicate findings to gain a better understanding of a phenomenon. Pointing to a single study as definitive proof for a viewpoint is naive and misguided.
[…] Education researcher Jay Greene has pointed out that not only did Ms. Phillips get the results of her own study wrong but she never bothered to correct the error publicly. Interest in the research or in winning the spin wars? Noble Foundation they are not. […]
“a lack of humility in the application of science to public policy”…yep, that would be Bill Gates…Mister “I will do whatever I damn well want to because,…frankly, I can buy you all!” In his crazed persuit to single handedly change the future of our free society, he would never let an insignificant thing like scientific method, statistical norms and validity or 100 years of prior sound psychological and educationational research stand in his way…It has nothing to do with ethical ‘best pratice”. It has everything to do with continuing to think and fund “outside of the box” cool ideas which will hand control of “the untapped market of public education” over from public control to private corporations. Just wait what he has in store for the future of computer technology replacing teachers with software in student cubicles K thru higher education. It’s costs are cheaper, so profits are greater. Bill Gate’s dream for the future of society. (Be very afraid. He is dead serious.)
Science also can’t really tell you what you should learn. That’s a real values question. I may prefer a classical school with essay tests only (even in Math). Emphasis on reading and writing. Works published before 1900. There’s no scientific answer to the question of whether or not that choice is “good”.