“Academics” and the “Practical” Part III: The Daleks Are Coming!

December 15, 2010

(Guest post by Greg Forster)

This week Jay highlighted the fact that a study showing a positive correlation between test prep activity in the classroom and improved learning is being portrayed as having shown a negative correlation between test prep activity (“drill and kill”) and improved learning. At this point it’s not known (at least to me) where the error arose, and I don’t have anything to say about the question of who said what. But I think it illustrates how the whole subject of drillandkillaphobia needs to be revisited.

Lately I’ve explored in some depth how testing has come to be the focal point of the fight between the two great factions in the education space, the “liberal artists” and the “pragmatists.” Liberal artists have gradually come to invest all their hopes in standardized testing. Pragmatists have gradually come to invest all their fears in precisely the same thing. This is a mutually reinforcing circle – over time, the liberal artists increasingly think testing must be good because the pragmatists hate it, and the pragmatists increasingly think it must be bad because the liberal artists love it. So naturally the battle line over testing has become more and more absolute.

The problem, as I’ve tried to show, is that this dynamic causes each side to reject something that’s essential to good education. The liberal artists seek curricula (or standards that can only lead to curricula) that emphasize sterile head knowledge of facts to the exclusion of practical problem-solving. We don’t know how to test what Daniel Willingham calls “deep knowledge” of subject area content, and even if we did the tests would be almost infinitely vulnerable to manipulation if they were ever used for accountability purposes. But we do know how to create rigorous tests for head knowledge of facts, so the liberal artists define “subject area content” to mean simply head knowledge of facts.

The pragmatists do the opposite. And since I was pushing pretty hard last time to emphasize what I thought were the dysfunctions of the liberal artists, I’d like to balance the scales with something about drillandkillaphobia.

Merely mention the subject of testing and it seems that pragmatists instantly jump to the conclusion that you want schools to look like this:

Now the interesting thing is that these days, they will engage in the most hysterical drillandkillaphobia while all the time affirming that we need to keep standards high, content knowledge matters, etc. To some extent I think that must be intentionally tactical – P21 knows that its flavor of loosey-goosey crunchy granola doesn’t sell these days, since they’ve lost a lot of battles in the war of ideas. But I don’t think that explains all of it. Take another look at that Ken Robinson “Changing Paradigms” video, where he begins by saying that of course we don’t want to lower standards. Clearly he, at least, really means it. It seems completely obvious to him that there’s no contradiction between his attacks on testing and his affirmation of high standards.

And that’s the problem. It seems so obvious to him that he fails to even take the question seriously. (That was one of the key points in Willingham’s stimulating critique of Robinson’s video.)

If the liberal artists need to get over their testophilia, the pragmatists need to get over their drillandkillaphobia. I’m not aware of any hard evidence linking test prep to worse outcomes. Sure, lots of people are really convinced that it must be the case, but that’s hardly a solid ground for making policy. (The floor is open in the comments section if you have some hard evidence you want to share.) And it’s not like this is a new question. I’ll admit that I haven’t done an exhaustive review of the research (again, if you have, the floor is open) but Jay and I conducted a study a while back showing that attaching rewards and penalties to a test doesn’t change the results; that would seem to speak right to the heart of drillandkillaphobia. This new Gates Foundation study, finding a positive correlation between test prep and learning outcomes, would seem to be another piece of evidence against it.

People can’t learn what Willingham calls the “deep structure” of practical problems until they’ve learned what he calls the “surface structure.” You can’t get from the pool deck to the bottom of the pool without passing through the surface of the water; similarly, you can’t get to deep (i.e. practical) content knowledge without first getting shallow (i.e. factual) content knowledge. If you like, it’s “merely” or “sterile” head knowledge. But head knowledge it is and head knowledge it will remain, even after you add the “deep” part.

People learn head knowledge by memorization. And any kind of memorization will appear, to those who wish to stigmatize and delegitimize it, to be “merely” “rote” memorization. You can call it “regurtitation” when people know facts, and in a sense you’re right – but people do need to know facts and be able to summon up that knowledge as necessary, whether you call that “regurgitating” or not. And for everyone but the real genius students, gaining head knowledge of facts will involve some kind of “drill.”

As the pragmatists themselves never tire of reminding us, real learning is hard. Well, yes it is. You can’t learn if you don’t memorize stuff. Memorizing stuff is hard and unpleasant, and it’s a lot more so for some kids than for others. That’s the world; deal with it.

I fully admit that if you really want to learn you are never just memorizing. You must be trying to understand the facts you absorb – understand their significance and the connections between them. But while it must be more than memorizing, it is never less than memorizing. Of course, if it must be more then by definition it can never be less.

Take that great, perennial boogyman of rote memorization – historical dates. People whine, why does it matter in what year a certain event occurred? Well – why does it matter? If you stopped and seriously asked that question and sought out the answer, you might . . . well, you might learn something.

The great irony, of course, is that at the same time the pragmatists are pushing this new bout of drillandkillaphobia, they’re working hard to impose a federal-government controlled system of national testing – excuse me, a totally “voluntary” system of “common” “assessment” that has nothing to do with the federal government, nothing to see here, move along, pay no attention to the man behind the curtain. That really is an attempt to handcuff real learning and turn us all over to the benevolent dictatorship of soulless men in white coats who must be trustworthy because, after all, they’re scientists. And that, of course, was always the original sin of the Progressive movement, in its educational form as in all other forms. Handing over all power to a tiny priesthood is the very epitome of “democracy” as long as we’re careful to call the priests “scientists.” But now I’m broaching a whole new and much deeper subject, one that will require another post to handle with any justice.


False Claim on Drill & Kill

December 13, 2010

The Gates Foundation is funding a $45 million project to improve measures of teacher effectiveness.  As part of that project, researchers are collecting information from two standardized tests as well as surveys administered to students and classroom observations captured by video cameras in the classrooms.  It’s a big project.

The initial round of results were reported last week with information from the student survey and standardized tests.  In particular, the report described the relationship between classroom practices, as observed by students, and value-added on the standardized tests.

The New York Times reported on these findings Friday and repeated the following strong claim:

But now some 20 states are overhauling their evaluation systems, and many policymakers involved in those efforts have been asking the Gates Foundation for suggestions on what measures of teacher effectiveness to use, said Vicki L. Phillips, a director of education at the foundation.

One notable early finding, Ms. Phillips said, is that teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains than those who simply work their way methodically through the key concepts of literacy and mathematics. (emphasis added)

I looked through the report for evidence that supported this claim and could not find it.  Instead, the report actually shows a positive correlation between student reports of “test prep” and value added on standardized tests, not a negative correlation as the statement above suggests.  (See for example Appendix 1 on p. 34.)

The statement “We spend a lot of time in this class practicing for [the state test]” has a correlation of  0.195 with the value added math results.  That is about the same relationship as “My teacher asks questions to be sure we are following along when s/he is teaching,” which is 0.198.  And both are positive.

It’s true that the correlation for “Getting ready for [the state test] takes a lot of time in our class” is weaker (0.103) than other items, but it is still positive.  That just means that test prep may contribute less to value added than other practices, but it does not support the claim that  “teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains…”

In fact, on page 24, the report clearly says that the relationship between test prep and value-added on standardized tests is weaker than other observed practices, but does not claim that the relationship is negative:

The five questions with the strongest pair-wise correlation with teacher value-added were: “Students in this class treat the teacher with respect.” (ρ=0.317), “My classmates behave the way my teacher wants them to.”(ρ=0.286), “Our class stays busy and doesn’t waste time.” (ρ=0.284), “In this class, we learn a lot almost every day.”(ρ=0.273), “In this class, we learn to correct our mistakes.” (ρ=0.264) These questions were part of the “control” and “challenge” indices. We also asked students about the amount of test preparation they did in the class. Ironically, reported test preparation was among the weakest predictors of gains on the state tests: “We spend a lot of time in this class practicing for the state test.” (ρ=0.195), “I have learned a lot this year about the state test.” (ρ=0.143), “Getting ready for the state test takes a lot of time in our class.” ( ρ=0.103)

I don’t know whether something got lost in the translation between the researchers and Gates education chief, Vicki Phillips, or between her and Sam Dillon at the New York Times, but the article contains a false claim that needs to be corrected before it is used to push changes in education policy and practice.

UPDATE —

The LA Times coverage of the report contains a similar misinterpretation: “But the study found that teachers whose students said they “taught to the test” were, on average, lower performers on value-added measures than their peers, not higher.”

Try this thought experiment with another observed practice to illustrate my point about how the results are being mis-reported…  The correlation between student observations that “My teacher seems to know if something is bothering me” and value added was .153, which was less than the .195 correlation for “We spend a lot of time in this class practicing for [the state test].”  According to the interpretation in the NYT and LA Times, it would be correct to say “teachers who care about student problems tend to have lower value-added learning gains than those who spend a lot of time on test prep.”

Of course, that’s not true.  Teachers caring about what is bothering students is positively associated with value added just as test prep is.  It is just that teachers caring is a little less strongly related than test prep.  Caring does not have a negative effect just because the correlation is lower than other observed behaviors.

(edited for typos)


Who’s to Blame for College Dropouts?

December 10, 2010

(Guest post by Greg Forster)

Education Week has some survey results that most people probably won’t find surprising or controversial, but that actually raise some tough ethical questions:

The public pins most of the blame for poor college graduation rates on students and their parents and gives a pass to colleges, government officials and others, a new Associated Press-Stanford University poll shows…

When asked where the blame lies for graduation rates at public four-year colleges, 7 in 10 said students shouldered either a great deal or a lot of it, and 45 percent felt that way about parents.

Others got off relatively easy: Anywhere between 25 percent and 32 percent of those polled blamed college administrators, professors, teachers, unions, state education officials and federal education officials.

Plausible enough! If you go to college and don’t graduate, whose fault is it but yours?

Yet Education Week thinks this is bad news for folks like us:

The belief that students are most at fault for graduation rates is a troubling sign for reformers who have elevated college completion to the forefront of higher education policy debates and pushed colleges to fix the problem, said Michael Kirst, professor emeritus of education and business administration at Stanford.

“The message is, ‘Students, you had your shot at college and failed and it’s your fault, not the college,'” Kirst said.

You know what? I think they’re right. This is bad news for us on a certain level. Gathering political capital for reforms that will do something about our collegiate dropout factories requires us to convince people that the colleges, not just the students, need to change.

And yet I don’t think the popular view is quite wrong. Check out this quote:

“We’re all responsible for our own education, and by the time you get to college you are definitely responsible and mature,” said Deanna Ginn, a mother of 12 from Fairbanks, Alaska.

Ginn is right. We don’t want to undermine the personal responsibility of each individual student who drops out.

Yet that can easily become an excuse to take colleges off the hook for their legitimate responsibilities, and these survey results show that happening.

Matt has spent a lot of time documenting the disaster in Arizona, where (to pick just a couple of many shocking numbers he’s posted here) ASU graduates only 28% of students in four years, and the University of Arizona 33%. But those are not really unusual numbers – this is not just an Arizona problem.

No doubt, as thousands of kids come in, spend money and then drop out, year after year, the college administrators and their hired protectors in state legislatures tell themselves exactly what Ginn says – if you drop out, it’s your own fault.

Well, yes. If you drop out, it is your own fault. But let’s think about this. Year after year, you see these dismal dropout rates. Is that really no business of yours? Is it really OK to say that sure, something like two-thirds of the new students we’re accepting this year are making a bad decision by coming here, but hey, as long as their tuition checks clear, that’s their headache, not ours?

I say if you run a college that has a two-thirds dropout rate, it’s your responsibility to do something about that. Taking people’s money to help them make bad choices is both morally wrong and, even from a merely selfish standpoint, imprudent in the long term. You should be asking how you can help ensure that kids who have a low likelihood of graduating are steered onto some alternate path – perhaps you can develop an alternate path for them at your own institution, which would allow you to cash their tuition checks and also look yourself in the mirror with respect. I don’t think that’s inconsistent with saying that each and every dropout is personally responsible for dropping out.

If I know you have an allergy to a vaccine that’s so bad you’ll die if I inject you with it, I have a responsibility not to inject you with it no matter how much money you offer me or how completely convinced you might be that it’ll do you good. After we say that, the principle also implies repsonsibilities at a little lower level than life and death.


Does Parent Trigger Cut the Gordian Knot?

December 8, 2010

(Guest post by Greg Forster)

The editorial in yesterday’s Journal covering the “parent trigger” earthquake in Los Angeles – at McKinley Elementary in Compton – argues that this could be a revolutionary new mechanism for advancing parental control of schools:

The biggest obstacle to education reform has long been overcoming the inertial forces of unionized bureaucracy. Parent trigger is a revolutionary shortcut, and bravo to the parents in Compton for making the leap.

The model is set to spread, argue the editors:

Parent trigger has support from Democrats including Sacramento Mayor Kevin Johnson, Los Angeles Mayor Antonio Villaraigosa, former Washington, D.C., schools chief Michelle Rhee and even Rahm Emanuel now that he’s running for mayor of Chicago. Legislators in Georgia, Indiana, Michigan, New Jersey, West Virginia and Maryland tell us they will introduce versions of parent trigger in the coming months.

Last time I looked in on the state of school governance reform in LA, I was skeptical. But that was more than a year ago, when the parent trigger mechanism wasn’t yet a part of the reform package. Last fall they were setting themselves up to have the public system hire private managers – which hasn’t worked in the past.

The parent trigger model is different. At a school that hasn’t made Adequate Yearly Progress ™ four years running, get a majority of parents to sign a petition and you can close the school, change administrators, or turn over the school to charter operators. The key difference is that the parents signing the petition decide what happens.

The district will fight them in court, of course, and they may win on a bogus technicality. As we learned in Florida in 2006, when the unions demand obeisance from their slaves you can’t count on a court to follow even the most tranparently clear meaning of the letter and spirit of the law.

But that’s not really relevant to the real policy question. All school reform policies are exposed to the naked assertion of thuggish power from union-bootlicking judges, and I don’t see much reason to think this one is more exposed (at least in principle) than any others.

So, that aside, is the Journal right that parent trigger is a way to cut the Gordian knot? Here are the advantages and disadvantages as I see them.

Advantages:

  1. School choice as a consequence of school failure is a proven way to improve public school performance. Even where the threat is never actualized, the mere threat produces clear gains.
  2. The parent trigger system may overcome the serious procedural obstacles that have dogged other “failing schools” models. The system for activating choice is (with an exception I’ll discuss below) simple, clear and not under the control of the government bureaucracy – and informing parents about their choices is easier because the system for creating choices involves getting parents informed and involved.
  3. The system is politically attractive, and partly for the right reasons. If a majority of the actual parents in the school want the school handed over, it’s really hard to be the people who say it shouldn’t be handed over.

Disadvantages:

  1. For the moment, the system is only promoting management change, at best involving charter operators, which is an improvement but is inadequate. But that’s less important because you could always use a parent trigger to activate vouchers.
  2. Petitions carry some problematic issues as a vehicle. Phrasing can be unclear, and/or people may not understand what they’re signing. Worse, the blob could organize its own counter-petitions to create confusion. It’s unlikely they could actually seize control of a school this way, but they could disrupt the process.
  3. More seriously, the system is only available at a small number of schools (those that don’t make AYP four years running). You could always fight to expand that, but the question is how far you could expand it. In theory you could do a parent trigger everywhere, but it’s not clear whether that would be politically viable. Maybe it would be if you did it in the right state. The larger question here is how wedded we are to a “failing schools” model that assumes schools are only failing if they’re populated by kids who are poor and dark-skinned. It’s an important question whether the parent trigger could be used to transition to a “failing schools” model that says any school repudiated by its parents is a failing school, or if it only reinforces the worst of our existing prejudices about what constitutes educational failure.
  4. Along a smiliar line, in its current form the parent trigger (like all previous “failing schools” models) reinforces government’s right to decide what constitutes a good education, because it relies on state testing as a parent-choice gatekeeper. In addition to my recent movement toward stronger critique of accountability testing for what are essentially pedagogical reasons, on an even more basic level it’s imperative that we not validate the idea that a good education is what government says it is. This, and #3 above, are what I meant when I said that parent trigger is politically attractive “partly” for the right reason. 
  5. Carrying on the theme of #3 and #4, most Americans wrongly believe there’s nothing wrong with their own schools; after all, the kids are middle-class whites and the schools are run by the government – nice, clean suburban government, not those icky urban machines – so how bad could they be? So suppose you give everyone a parent trigger and don’t get enough schools where you overcome all the obstacles of perception (to say nothing of the logistics) and get a majority to sign off. That would only validate the illusion that the status quo in the great suburban Middle America is A-OK.

So color me ambivalent. Parent trigger is certainly an improvement over Florida’s A+ model, where near-insuperable bureaucratic obstacles stood between parents and the actual excercise of choice. And I see some potential to use this as a path to making parents’ judgments the standard for what counts as a good school. But there are serious dangers here as well, if we don’t take seriously the omnipresent temptation to slide back toward liberal paternalism.


Finland Sucks

December 7, 2010

Actually, I don’t really think so.  But if I were Diane Ravitch and looked at the trend in PISA for Finland as she looked at the trend in NAEP for New York City, I would see that Finland has declined in reading, math, and science.  And then I would (wrongly) conclude that Finland sucks and is doing things all wrong.

Table 5.1 Finland’s mean scores on reading, mathematics and science scales in PISA (p. 118)

PISA 2000 PISA 2003 PISA 2006 PISA 2009
Mean score Mean score Mean score Mean score
Reading 546 543 547 536
Mathematics 544 548 541
Science 563 554

Or perhaps if I really wanted to be like Diane Ravitch I would switch from looking at trends to levels of achievement, like when she looks at Massachusetts.  In that case, I would still think Finland is great and doing everything right.

Or maybe I could be like Diane Ravitch and switch to a different test that produced results more to my liking, like when Diane stopped paying attention to NAEP for New York City when it showed significant gains and started focusing instead on problems in the state test measures.

That’s the problem with being a manipulative propagandist.  It’s so hard to keep your story straight from one deception to another.


Supermarket Fail

December 7, 2010

It’s not just schools.  All organizations struggle to operate efficiently.  The only difference is that some organizations experience consequences when they fail and others don’t.

(HT: LW)


Chris Christie Kicks Butt at Jebfest 2010

December 7, 2010

You really have to check this out.  Christie has an intensity and directness about the problems with teacher unions and the educational status quo that is wonderfully refreshing and unfortunately too rare among politicians.

(You have to click the link in the first sentence above to go to CSPAN and watch the video. I can’t seem to embed the video.   CSPAN and WordPress don’t seem to get along.)


Burke and Ladner respond to the Think Tank Review Project

December 6, 2010

(Guest Post by Matthew Ladner)

Lindsey Burke and I respond to a critique of our work by the Think Tank Review project. Perhaps my favorite part is where the reviewer chides us for failing to do a literature review but failed to notice that Dan Lips and I had refuted her main contention over a year ago in Education Next. The main point of the review is an attempt to cry foul over Florida’s 4th Grade NAEP scores due to the 3rd grade retention policy.

I’ve changed my mind, this is my favorite part:



This page comes directly from the review. Notice that by the information gathered by the reviewer herself, the percentage of Florida students scoring FCAT 1 (the lowest possible score and the score making a student eligible for retention) in 3rd grade fell from 27% in 2001-02 to 17% in 2008-09. That’s almost a 38% decline.

Likewise, the percentage of African-American students scoring FCAT 1 fell from 41% to 27%, and the Hispanic rate fell from 35% to 21%. Notice that the African-American rate of scoring FCAT 1 now matches the overall rate in 2001-02 had been.

This is called “radical success.”

Notice also that the number of students actually being retained drops by more than half between the first year of the policy and 2008-09. Despite that fact, Florida’s 4th grade NAEP scores continued to climb. If Florida’s NAEP improvement were driven by retention, scores ought to have peaked early in the decade, and then fallen off. Instead, they continued to rise throughout the decade, even as retention declined.

Oh, and Florida’s reading scores improved by almost a grade level before the retention policy even passed. I could go on, point out the practice of mid-year promotions further weakens the “it was all retentions” theory, and/or blather at some length about the regression discontinuity analysis that Jay performed, which strongly points to something other than aging going on with this policy. Click the link if you want to read about it.

The bottom line: these policies worked. The percentage of Florida students scoring below basic on 4th grade reading dropped from 47% to 27% between 1998 and 2009. No one knows exactly how much of which policy moved the needle, but there is a simple solution to this: do all of the policies at the same time.

If Florida lawmakers had mandated in 1999 that students stood off the side of their desks to do jumping jacks to start each school day, and childhood illiteracy dropped like a rock off a cliff, I would be advocating for other states to do the same. At least until such time that someone established that it didn’t add any value.

In some offline conversations I have had with the Think Tank Review people, they seem to think that other states should be “cautious” until we know exactly how much improvement there has been after hair-splitting, and what causes what.

I disagree. In my view, that’s like getting into an argument about whether to use the sprinkler system, the firehose or the buckets of water when kids are running around with their hair on fire. Florida used all the approaches at once, and got great improvement.

Governor-elect Scott seems to busily readying Florida Reform Version 2.0. Somehow I doubt he will be much persuaded by an attempt to muddy the water on Version 1.0.

All is not lost, however.

I will be adding the above table from their study to my Powerpoint, given how well it makes the case for Florida’s reforms.


LA Times Piece Documents Harms of Seniority-Based Layoffs

December 6, 2010

The LA Times has another great article based on the analysis they have conducted with a RAND researcher on the value-added of LA public school teachers.  This one shows that seniority-based lay-offs, as required by many union contracts, are hurting kids:

The Times sought to measure the impact of about 2,700 seniority-based layoffs in the Los Angeles Unified School District in the last two years. It focused particularly on the performance of about 1,000 elementary and middle school teachers for whom math and English scores were available.

Among the findings:

Because seniority is largely unrelated to performance, the district has laid off hundreds of its most promising math and English teachers. About 190 ranked in the top fifth in raising scores and more than 400 ranked in the top 40%.

Schools in some of the city’s poorest areas were disproportionately hurt by the layoffs. Nearly one in 10 teachers in South Los Angeles schools was laid off, nearly twice the rate in other areas. Sixteen schools lost at least a fourth of their teachers, all but one of them in South or Central Los Angeles.

Far fewer teachers would be laid off if the district were to base the cuts on performance rather than seniority. The least experienced teachers also are the lowest-paid, so more must be laid off to meet budgetary targets. An estimated 25% more teachers would have kept their jobs if L.A. Unified had based its cuts on teachers’ records in improving test scores.


Klein’s Lessons

December 6, 2010

*** Joel Klein tells us in the pages of the WSJ what he learned as Chancellor of NYC schools.  Here’s a highlight:

First, it is wrong to assert that students’ poverty and family circumstances severely limit their educational potential. It’s now proven that a child who does poorly with one teacher could have done very well with another. Take Harlem Success Academy, a charter school with all minority, mostly high-poverty students admitted by lottery. It performs as well as our gifted and talented schools that admit kids based solely on demanding tests. We also have many new small high schools that replaced large failing ones, and are now getting outsized results for poor children.

Second, traditional proposals for improving education—more money, better curriculum, smaller classes, etc.—aren’t going to get the job done. Public education is a service-delivery challenge, and it must be operated as such.

Klein raises an excellent point.  Diane Ravitch, Sol Stern, and others who claim that they have grown frustrated with choice and other incentive-based reforms because they haven’t yet produced the miracles they expected ought to be 1,000 times more frustrated with the failure of more money, higher teacher certification requirements, curricular and pedagogical reform, etc…  We’ve tried those kinds of reforms more than 1,000 times more on a much grander scale and yet we still wait for the miracles.

To judge the effectiveness of reform strategies we can’t use miraculous improvement as the standard.  And we certainly need more fine-grained analyses than looking at whether cities or states that have tried something have improved.  And finally, we can work on various types of reforms simultaneously, so pitting incentives versus instruction is a false conflict that serves only to inflate the ego-starved reformer rather than the cause of reform.