The Gates Foundation is funding a $45 million project to improve measures of teacher effectiveness. As part of that project, researchers are collecting information from two standardized tests as well as surveys administered to students and classroom observations captured by video cameras in the classrooms. It’s a big project.
The initial round of results were reported last week with information from the student survey and standardized tests. In particular, the report described the relationship between classroom practices, as observed by students, and value-added on the standardized tests.
The New York Times reported on these findings Friday and repeated the following strong claim:
But now some 20 states are overhauling their evaluation systems, and many policymakers involved in those efforts have been asking the Gates Foundation for suggestions on what measures of teacher effectiveness to use, said Vicki L. Phillips, a director of education at the foundation.
One notable early finding, Ms. Phillips said, is that teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains than those who simply work their way methodically through the key concepts of literacy and mathematics. (emphasis added)
I looked through the report for evidence that supported this claim and could not find it. Instead, the report actually shows a positive correlation between student reports of “test prep” and value added on standardized tests, not a negative correlation as the statement above suggests. (See for example Appendix 1 on p. 34.)
The statement “We spend a lot of time in this class practicing for [the state test]” has a correlation of 0.195 with the value added math results. That is about the same relationship as “My teacher asks questions to be sure we are following along when s/he is teaching,” which is 0.198. And both are positive.
It’s true that the correlation for “Getting ready for [the state test] takes a lot of time in our class” is weaker (0.103) than other items, but it is still positive. That just means that test prep may contribute less to value added than other practices, but it does not support the claim that “teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains…”
In fact, on page 24, the report clearly says that the relationship between test prep and value-added on standardized tests is weaker than other observed practices, but does not claim that the relationship is negative:
The five questions with the strongest pair-wise correlation with teacher value-added were: “Students in this class treat the teacher with respect.” (ρ=0.317), “My classmates behave the way my teacher wants them to.”(ρ=0.286), “Our class stays busy and doesn’t waste time.” (ρ=0.284), “In this class, we learn a lot almost every day.”(ρ=0.273), “In this class, we learn to correct our mistakes.” (ρ=0.264) These questions were part of the “control” and “challenge” indices. We also asked students about the amount of test preparation they did in the class. Ironically, reported test preparation was among the weakest predictors of gains on the state tests: “We spend a lot of time in this class practicing for the state test.” (ρ=0.195), “I have learned a lot this year about the state test.” (ρ=0.143), “Getting ready for the state test takes a lot of time in our class.” ( ρ=0.103)
UPDATE —
The LA Times coverage of the report contains a similar misinterpretation: “But the study found that teachers whose students said they “taught to the test” were, on average, lower performers on value-added measures than their peers, not higher.”
Try this thought experiment with another observed practice to illustrate my point about how the results are being mis-reported… The correlation between student observations that “My teacher seems to know if something is bothering me” and value added was .153, which was less than the .195 correlation for “We spend a lot of time in this class practicing for [the state test].” According to the interpretation in the NYT and LA Times, it would be correct to say “teachers who care about student problems tend to have lower value-added learning gains than those who spend a lot of time on test prep.”
Of course, that’s not true. Teachers caring about what is bothering students is positively associated with value added just as test prep is. It is just that teachers caring is a little less strongly related than test prep. Caring does not have a negative effect just because the correlation is lower than other observed behaviors.
(edited for typos)
Jay P. Greene's Blog 
Very interesting.
I have argued that classroom management skills are THE gateway skill for teachers in high-poverty schools, yet typically Ed Schools shy away from teaching them (part b/c they don’t know; part b/c “control” doesn’t square with their idealized view of teaching).
Do you think the Gates data supports my thesis?
The LA Times article contradicts the NY Times:
Also, value-added was found to be a reliable predictor of students’ future performance on different tests that measure higher-level concepts, particularly in math. “Teachers who are producing gains on the state tests are generally also promoting deeper conceptual understanding among their students,” the researchers found.
Many asian students achieve success in math by doing drills. With Kumon math, students K-3rd grade do about 50,000 repetitive math equations per year–compared to about 5,000 for American kids. The Kumon drills take about 15 minuites a day and have nothing to do with the regular math curriculum. What drills are american educators referring too? The dismal results in America prove that American educators don’t know how to write proper math drills. It’s the quality of the drill that matters and american educators don’t know or don’t want to know about Kumon. Research done by universities show that American teachers don’t want to deal with the large amounts of paperwork that result from drills, so they place a negative label “drill and kill” on the most useful aspect of early childhood math.