“Academics” and the “Practical” Part IV: Seizing Power

January 19, 2011

(Guest post by Greg Forster)

For a while now I’ve anticipated that the next installment of this series would be about power. Since Jay has broached the subject, I guess it’s finally time to get around to writing what I’ve been planning!

A quick review of my Unified Field Theorem of Education Reform:

Part 1: Education reformers shake out into two groups, which I call the “liberal artists” and the “pragmatists.” The liberal artists want to teach first the three Rs, then traditional “academic” content more generally. Their strength is their insistence on tangible accountability for teaching all children; their weakness is their overreliance on standardized testing – now culminating in the current effort to create a government-controlled national testing regime which logically implies the further step of imposing a single curriculum on all schools under the control of a central authority. The pragmatists want to make education in various ways “more relevant to real life.” Their strength is their desire to create new models of education that will prepare students better for life in the (changing) world. Their weakness is their tendency to discount (in practice if not in rhetoric) the value of traditional academics, and especially their fear of accountability systems.

Part 2: Both sides undermine not only education governance (the focus in Part 1) but pedagogy as well. The pragmatists want to abstract “skills” from “content” and focus on teaching the skills; they fail to appreciate that the only way to learn skills is by leaning content. You can never teach “skills” directly. The liberal artists want to abstract “knowledge” from “practice” and focus on teaching the knowledge; they fail to appreciate that all the really important knowledge is intricately bound up with practice, and can only be learned practically.

Part 3: Liberal artists need to get over their testaphilia, and pragmatists need to get over their testaphobia. A vast quantity of what students deperately need to learn must be learned in ways that can’t be tested with the level of objective systematization the liberal artists insist upon. You can “test” practical knowledge but not in the ways the liberal artists want – and not in ways that can be effectively used as the basis of an accountability regime. Yet standardized testing, and more generally the “rote” “regurgitation” of “mere” “facts,” is always going to be a crucial part of good education. In Daniel Willingham’s language, you can’t get to the “deep structure” of problems, which is what the pragmatists want, until you’ve first mastered the “surface structure,” which is the rote facts the pragmatists disdain. You have to walk before you can fly.

Running through all this is the tension between governance and pedagogy. Once we decide what we want schools to do, how do we structure the system to try to get them to do that?

The links beteween each camp’s pedagogy and governance, both in their good and bad aspects, run much deeper than it may at first appear. What’s really at stake is our view of the human person.

An analogy to politics will help here. In modern political philosophy there are basically three anthroplogies on offer. They give rise to different political systems.

  • You can be cynical about human nature, thinking that people are basically bad. This leads more or less directly to an explicitly authoritarian, implicitly totalitarian tyranny of “enlightened” despots. Because people are basically bad, no one can ever have legitimate power (no one deserves it) and the world will really operate by illegitimate power no matter what you do. So you might as well give the power to the smartest people so they will at least make things run more smoothly and everyone will have an easier time of it. Machiavelli and Hobbes fit this model.
  • You can be naive about human nature, thinking that people are basically good. This also leads more or less directly to an explicitly authoritarian, implicitly totalitarian tyranny of “enlightened” despots. Because people are good, they will naturally want to cooperate to make everyone better off, which of course means putting things under the control of the smartest, best people. And those people ought to have the power to coerce everyone’s cooperation, becasue such power won’t really need to be exercised very much – just enough to encourage people to get over their less powerful selfish tendencies and live into their natural desire to benefit others, which is (underneath the superficial layer of selfishness) really their deeper and stronger desire. The payoff from giving dictatorial power to experts is huge (because the experts are not only smart but good and trustworthy) and the cost is small (because the power won’t have to be exercised much). Rousseau and Hegel fit this model.
  • You can take a mixed view of human nature, thinking that people are both basically good and basically bad. They need freedom to do their good stuff, but also enough restraint to keep them from getting out of line and destroying other people’s good stuff; the rulers, in turn, must be strong enough to restrain violence, but not so strong that they themselves become unaccountable. This is the anthropology of liberal democracy, freedom of religion, and the entrepreneurial economy; Locke, Montesquieu and Madison are its architechts.

The thing to note is that societies cannot be counted on to remain faithful to one model. In particular, the mixed model on which liberal democracy, freedom of religion and entrepreneurial economy are built is really darned difficult to maintain. We are constantly falling away into cynicism on the one side (e.g. Cass Sunstein, Catherine MacKinnon, Saul Alinsky) or naivete on the other (e.g. Michael Lerner, Alan Wolfe, Jim Wallis) with the same disastrous consequences every time.

How does this relate to pedagogy and governance in education? I propose that education needs to be based on a mixed model, but is constantly falling away into one or the other of two truncated models – and that’s why substantive education issues are constantly being hijacked by brute political power.

Look at the liberal artists. How did we get to a point where the people dedicated to the full flowering of human knowledge represented by “traditional academics” are in the process of reducing the content of education to what can be measured by bubble tests – and lining up to create a national dictator that will reach into every school in America and crush everything that isn’t bubble tests?

It’s because their anthropology privileges intellect over action. A human being is a mind that has a body. What they want is to educate the mind. The body is really of no concern to them. Even the mind is only of interest insofar as it knows things – the mind’s ability to do things through the body is not interesting. Re-reading my first post in this series, this is really clear in the exchange between Jay and Checker about whether schools should teach things like “entrepreneurial attitudes.”

Checker Finn’s ideal school

This anthropology implies an aristocracy of intellect. The system should serve the interests of those who are capable of learning. The liberal artists think they’re egalitarians and democratizers because they stick up for the poor black kids who want to learn – and, as I have said over and over, they’ve done us a great service. They have indeed been the great titanic warriors against race and class aristocracies. But there are other kinds of aristocracies as well. The liberal artists only stick up for the kids who want to learn in a certain way: the intellectual way, the bubble test way. They want the whole system to serve only the kids who desire to know for knowledge’s sake – and that’s not most kids. What about the kids who want to invent new things, or acheive greatness in other ways, and who might be willing to learn academics as a stepping stone to that but not for its own sake? They’re chucked into the maw of the intellectual tyranny.

Ken Robinson was wrong (in that video back in Part 1) to attribute this anthropology to the Enlightenment; it is actually far older, and has historically been associated with undemocratic power structures. Mind/body dualism was the philosophy of Greco-Roman aristocracy – Athens was the only democracy of any importance in the ancient world, and it executed the great dualist Socrates. Even during the Enlightenment, those who strongly embraced mind/body dualism (like Descartes) were strong supporters of traditional power structures. It was those who challenged mind/body dualism, like Locke, who ushered in democracy.

What about the pragmatists? It’s tempting to say that they have the opposite problem – they think kids are bodies that have minds. But that’s actually wrong. The strength of the pragmatists is that they’re not plagued by mind/body dualism.

Their problem is egalitarianism. They don’t want education to result in inequalities – no inequalities of life outcomes, but more fundamentally, no inequalities of educational outcomes. To draw a distinction, even in thought, between those who can accomplish more and those who can accomplish less is itself wrong. Anything that tends to reinforce the appearance of such distinctions, or (worse) explicitly assume such distinctions and build on them, is in principle radically evil.

This explains their testaphobia and their general aversion to accountability systems. It also explains why they are de facto but not de jure hostile to traditional academics. They have no objection to academics in principle – provided the illusion of equal outcomes is not punctured. But, of course, it always is. Much safer to stick to content-free, purely “practical” projects that teach “skills.”

The picture offered to us is one of glorious diversity in which every child is radically different, and none of the differences matter.

“You can think for yourselves!”
“Yes! We can think for ourselves!”
“You are all individuals!”
“Yes! We are all individuals!”

Naturally, while the liberal artists are striving to build an aristocracy, the pragmatists are striving to build a tyranny of the majority – the mob rule of unlimited democracy. This was their original sin going all the way back to John Dewey, whose perfidy begins right at the beginning when he sets out to redesign the whole educational enterprise to produce, not the fullest possible flourishing of human capacities, but people suited to fit the new, radical political system that early 20th century progressives were working so hard to build. Human beings are little clay figures just waiting for Dewey and his acolytes to mold them into the politically convenient shape. All the worst aspects of educational pragmatism can really be traced back to this original politicization of the project.

So both camps find substantial resistance to their desires – the liberal artists, in children who are capable of achievement but don’t highly value knowledge for its own sake; the pragmatists, in children who are capable of excellence (even the kinds of excellence pragmatists claim to value) and need special nurturing to achieve it.

And both camps, their vision cramped by narrow anthropologies, fail to see the legitimacy of this resistance. To them, the resistance appears to be simply obscurantism. Hence they feel perfectly justified stamping it out by force.

And naturally, when they seek power, they both reach for the strongest of all social weapons in modern culture – science.

Say that you favor a given approach – in education, in politics, in culture – because it is best suited to the nature of the human person, or because it best embodies the principles and historic self-understanding of the American people, and you will struggle even to get a hearing. But if you say that “the science” supports your view, the world will fall at your feet.

Of course, this means powerful interest groups rush in to seize hold of “science,” to trumpet whatever suits their preferences, downplay its limitations, and delegitimize any contrary evidence. If they succeed – which they don’t always, but they do often enough – “the science” quickly ceasees to be science at all. That’s why “scientific” tyrannies like the Soviet Union had to put so many real scientists in jail – or in the ground.

This was, again, the original sin of Dewey and the whole “pragmatist” movement in early 20th century philosophy. The goal of that school was to undermine the philosophical structure of knowledge on which real science depends, so that they would then have a free hand to bend “science” to their will. I believe it was George Orwell who said that philosophical pragmatism amounts to saying that truth is determined by who has more guns.

But the liberal artists are no longer very much better. They didn’t used to flatten their understanding of a good education down to the level that could be measured “scientifically” on tests. But the imperative to seek power and crush resistance has driven them to that point.

How, then, do we escape from both aristocracy and mobocracy, and undo the tyranny of science? Stay tuned.


More on Scientific Progressivism

January 19, 2011

I just wanted to add a few thoughts to my post yesterday.  Readers may be wondering what is wrong with using science to identify the best educational practices and then implementing those best practices.  If they are best, why wouldn’t we want to do them?

Let me answer by analogy.  We could use science to identify where we could get the highest return on capital.  If science can tell us where the highest returns can be found, why would we want to let markets allocate capital and potentially make a lot of mistakes?  Government could just use science and avoid all of those errors by making sure capital went to where it could best be used.

Of course, we tried this approach in the Soviet Union and it failed miserably.  The primary problem is that science is always uncertain and susceptible to corruption.  We can run models to measure returns on capital, but we have uncertainty about the models and we have uncertainty about the future.  Markets provide a reality test to scientific models by allowing us to choose among competing models and experience the consequences of choosing wisely or not.  Science can advise us, but only choice, freedom, and experience permit us to benefit from what science has to offer.

And even more dangerous is that in the absence of choice and competition among scientific models, authorities will allow their own interests or preferences to distort what they claim science has to say.  For an excellent example of this, check out the story of Lysenko and Soviet research on genetics.  For decades Soviet science was compelled to believe that environmental influences could be inherited.

Science facilitates progress through the crucible of market tests.  Science without markets facilitates stronger authoritarianism.


The Dead End of Scientific Progressivism

January 18, 2011

In Education Myths I argued that we needed to rely on science rather than our direct experience to identify effective policies.  Our eyes can mislead us, while scientific evidence has the systematic rigor to guide us more accurately.

That’s true, but I am now more aware of the opposite failing — believing that we can resolve all policy disputes and identify the “right way” to educate all children solely by relying on science.  Science has its limits.  Science cannot adjudicate among the competing values that might attract us to one educational approach over another.  Science usually tells us about outcomes for the typical or average student and cannot easily tell us about what is most effective for individual students with diverse needs.  Science is slow and uncertain, while policy and practice decisions have to be made right now whether a consensus of scientific evidence exists or not.  We should rely on science when we can but we also need to be humble about what science can and can’t address.

I was thinking about this while reflecting on the Gates Foundation’s Measuring Effective Teachers Project.  The project is an ambitious $45 million enterprise to improve the stability of value-added measures while identifying effective practices that contribute to higher value-added performance.  These are worthy goals.  The project intends to advance those goals by administering two standardized tests to students in 8 different school systems, surveying the students, and videotaping classroom lessons.

The idea is to see if combining information from the tests, survey, and classroom observations could produce more stable measures of teacher contributions to learning than is possible by just using the state test.  And since they are observing classrooms and surveying students, they can also identify certain teacher practices and techniques that might be associated with greater improvement.  The Gates folks are using science to improve the measures of student progress and to identify what makes a more effective teacher.

This is a great use of science, but there are limits to what we can expect.  When identifying practices that are more effective, we have to remember that this is just more effective for the typical student.  Different practices may be more effective for different students.  In principle science could help address this also, but even this study, with 3,000 teachers, is not nearly large enough to produce a fine-grained analysis of what kind of approach is most effective for many different kinds of kids.

My fear is that the researchers, their foundation-backers, and most-importantly, the policymaker and educator consumers of the research are insensitive to these limitations of science.  I fear that the project will identify the “right” way to teach and then it will be used to enforce that right way on everyone, even though it is highly likely that there are different “right” ways for different kids.

We already have a taste of this from the preliminary report that Gates issued last month.  Following its release Vicki Phillips, the head of education at the Gates Foundation, told the New York Times: “Teaching to the test makes your students do worse on the tests.”  Science had produced its answer — teachers should stop teaching to the test, stop drill and kill, and stop test prep (which the Gates officials and reporters used as interchangeable terms).

Unfortunately, Vicki Phillips mis-read her own Foundation’s report.  On p. 34 the correlation between test prep and value-added is positive, not negative.  If the study shows any relationship between test prep and student progress, it is that test prep contributes to higher value-added.  Let’s leave aside the fact that these were simply a series of pairwise correlations and not the sort of multivariate analysis that you would expect if you were really trying to identify effective teaching practices.  Vicki Phillips was just plain wrong in what she said.  Even worse, despite having the error pointed out, neither the Gates Foundation nor the New York Times has considered it worthwhile to post a public  correction.  Science says what I say it says.

And this is the greatest danger of a lack of humility in the application of science to public policy.  Science can be corrupted so that it simply becomes a shield disguising the policy preferences of those in authority.  How many times have you heard a school official justify a particular policy by saying that it is supported by research when in fact no such research exists?  This (mis)use of science is a way for authority figures to tell their critics, “shut up!”

But even if the Gates report had conducted multivariate analyses on effective teaching practices and even if Vicki Phillips could accurately describe the results of those analyses, the Gates project of using science to identify the “best” practices is doomed to failure.  The very nature of education is that different techniques are more effective in different kinds of situations for different kinds of kids.  Science can identify the best approach for the average student but it cannot identify the best approach for each individual student.  And if students are highly varied in their needs, which I believe they are, this is a major limitation.

But as the Gates Foundation pushes national standards with new national tests, they seem inclined to impose the “best” practices that science identified on all students.  The combination of Gates building a national infrastructure for driving educator behavior while launching a gigantic scientific effort to identify the best practices is worrisome.

There is nothing wrong with using science to inform local practice.  But science needs markets to keep it honest.  If competing educators can be informed by science, then they can pick among competing claims about what science tells us.  And they can learn from their experience whether the practices that are recommended for the typical student by science work in the particular circumstances in which they are operating.

But if the science of best educator practice is combined with a national infrastructure of standards and testing, then local actors cannot adjudicate among competing claims about what science says.  What the central authorities decide science says will be infused in the national standards and tests and all must adhere to that vision if they wish to excel along these centralized criteria.  Even if the central authority completely misunderstands what science has to say, we will all have to accept that interpretation.

I don’t mean to be overly alarmist.  Gates has a lot of sensible people working for them and there are many barriers remaining before we fully implement national standards and testing.  My concern is that the Gates Foundation is being informed by an incorrect theory of reform.  Reform does not come from science identifying the right thing to do and then a centralized authority imposing that right thing on everyone.  Progress comes from decentralized decision-makers having the freedom and motivation to choose among competing claims about what is right according to science.

(edited for typos)


Government Takeover of STEM

January 4, 2011

(Guest post by Greg Forster)

In his Sunday column, George Will advocated a government takeover of the economy. Well, not quite – but close.

Will points out, correctly, that the economy is really, ultimately driven by the discovery of new ways of serving human needs. From this, he concludes that the enormous government regime of subsidies (and consequent control) of “basic science” and other STEM (science, technology, engineering and math) research in universities must not only continue, but be dramatically expanded.

He makes the by-now standard argument for government control of STEM:

  1. STEM contributes to the economy through “basic science.”
  2. “Basic science” doesn’t yield useable results rapidly enough to show up on quarterly earnings reports.
  3. Businesses are incapable of seeing past quarterly earnings reports when making decisions.
  4. Therefore, only government (through its hired retainers in the universities) has a long enough time horizon to be entrusted with control of basic science, and hence STEM.

How is this wrong? Let me count the ways.

The error starts right at the beginning. “Basic science” is not what drives entrepreneurial innovation and economic flourishing. “Basic science” is part of the liberal arts and is not all that much different from the study of poetry. It’s about investigating the fundamental structure of the universe simply for the sake of understanding it – just like poetry, in a different way, investigates the fundamental structure of the universe simply for the sake of understanding it. Basic science not only doesn’t produce economic benefits on a quarterly basis, it doesn’t produce economic benefits at all (except insofar as it contributes generally to the maintenance of a humane culture). 

This matters because it is universities who fundamentally drive “basic science,” but not entrepreneurial innovation. There’s a reason Bill Gates had to leave Harvard to found Microsoft.

It’s entrepreneurially minded businesses that drive entrepreneurial innovation and economic flourishing. The assertion that businesses don’t support long-term innovation is false. Some do, some don’t. The ones that do are where the dynamism of the economy comes from. Google encourages employees to spend a set portion of their time working on side projects over which they have total control, and which are not expected to produce defined results; the Google News service was created as one such project. Yes, there are many businesses that can’t see past their quarterly earnings reports. But the solution to that is for a partnership of philanthropy and educational institutions to raise up a new generation of entrepreneurial leaders who can see past their quarterly earnings reports.

If business as a sector is congenitally and permanently incapable of long-term thinking, the United States is scrod, and we should all quit trying to save it.

The worst error is to think that government and universities are capable of better long-term strategic thinking than business. The opposite is the case. Just look at the outstanding examples of long-term strategic thinking we have before us in those sectors today – in government, $14 trillion debt with bailouts, nationalizations and endless Keynesianism (on the right and left) at home, and fecklessness and appeasement abroad; in the universities, a ridiculously unsustainable business model, the most dysfunctional labor policy (tenure) of any sector of society, and a total abandonment of the sector’s core function (education for human life) in favor of hyperspecialization of technical competencies.

The main difference between business and the government/university axis is that business occasionally does really take care for the long term.


Fordham Fears the Daleks

December 17, 2010

(Guest post by Greg Forster)

The Fordham Institute’s Gadfly, in an item signed by Janie Scull, picks up and recirculates the Gates Foundation’s drillandkillaphobic error – but with a subtle twist (see if you can catch it – I didn’t when I first read it):

Second, teachers who, according to their students, “teach to the test” do not produce the highest value-added scores for said students; rather, instructors who help their students understand math concepts and reading comprehension yield the highest scores.

Gates was originally pushing the line that the study found a negative relationship between “teach to the test” or “drill and kill” and outcomes. Not only is there nothing in the study to support that, the study actually finds the opposite.

Fordham is now slightly changing the claim so that it appears to say test prep is bad for students, without actually saying that. Read that sentence very carefully. Now read it again, and this time bear the following in mind: the study found a positive correlation between test prep and outcomes. Now, does this look like an honest characterization of the study to you?

I’ll admit that they had me fooled. I originally put up a version of this blog post saying that they were recirculating Gates’s erorr. Then I reread it and pulled that down. They’re not just recirculating the error, they’re weaseling it up to see if they can circulate it in a way that will pass muster. I haven’t seen such word-twisting since I watched the president of the United States explain “it depends on what the meaning of ‘is’ is.”

“I did not have a statistically significant relationship with that variable.”

Despite her efforts to remain technically correct, Scull does make an erroneous claim. She directly attributes the phrase “teach to the test,” in quotes, to students. In fact, as Jay pointed out, the phrase “teach to the test” and similar phobic phrases such as “drill and kill” do not appear in the study. The study found a positive relationship between “test prep” and outcomes.

This is worse than the New York Times and Los Angeles Times reporting the original error. Those papers simply picked up what Gates told them and reported that Gates said it. Sure, in a perfect world reporters would always check these things with independent researchers – but it’s not a hanging offense. (However comically hysterical some of them might get when they get called on it.)

The Fordham Institute is, or at least claims to be, an independent voice. And the Gadfly item did not attribute its claim to Gates, as the newspapers did. The Gadfly item states its partly erroneous, partly weaseled-up claim simply as a fact. That lends the intellectual prestige of the Fordham Institute to both the error and the Clintonian weaseling.

Jay has said before, and I agree, that Fordham can take huge piles of money from Gates without losing its integrity.

That’s why I have full confidence Scull and Fordham will be running a correction of this erroneous item.

As I wrote earlier this week, human beings are not daleks, so test prep and similar activities can’t be the be-all and end-all, but the fear of test prep has so far been much more destructive than its overemphasis. If we don’t get past drillandkillaphobia, we’ll never fix education.

[This post has been edited since it was first published, as indicated in the text.]


Drill and Kill Kerfuffle

December 16, 2010

The reaction of New York Times reporter, Sam Dillon, and LA Times reporter, Jason Felch,  to my post on Monday about erroneous claims in their coverage of a new Gates report could not have been more different.  Felch said he would look into the issue, discovered that the claimed negative relationship between test prep and value-added was inaccurate, and is now working on a correction with his editors.

Sam Dillon took a very different tack.  His reaction was to believe that the blog post was “suggesting on the internet that I had misinterpreted an interview, and then you repeated the same thing about the Los Angeles Times. That was just a sloppy and irresponsible error.”  I’m not sure how Dillon jumps to this thin-skinned defensiveness when I clearly said I did not know where the error was made: “I don’t know whether something got lost in the translation between the researchers and Gates education chief, Vicki Phillips, or between her and Sam Dillon at the New York Times, but the article contains a false claim that needs to be corrected before it is used to push changes in education policy and practice.

But more importantly, Dillon failed to check the accuracy of the disputed claim with independent experts.  Instead, he simply reconfirmed the claim with Gates officials: “For your information, I contacted the Gates Foundation after our correspondence and asked them if I had misquoted or in any way misinterpreted either Vicki Phillips, or their report on their research. They said, ‘absolutely not, you got it exactly right.'”

He went on to call my efforts to correct the claim “pathetic, sloppy, and lazy, and by the way an insult.”  I guess Dillon thinks that being a reporter for the New York Times means never having to say you’re sorry — or consult independent experts to resolve a disputed claim.

If Dillon wasn’t going to check with independent experts, I decided that I should — just to make sure that I was right in saying that the claims in the NYT and LAT coverage were unsupported by the findings in the Gates report.

Just to review, here is what Dillon wrote in the New York Times: “One notable early finding, Ms. Phillips said, is that teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains than those who simply work their way methodically through the key concepts of literacy and mathematics.”  And here is what Jason Felch wrote in the LA Times: ““But the study found that teachers whose students said they ‘taught to the test’ were, on average, lower performers on value-added measures than their peers, not higher.”  And the correlations in the Gates report between test student reports of test prep and value-added on standardized tests were all positive: “We spend a lot of time in this class practicing for the state test.” (ρ=0.195), “I have learned a lot this year about the state test.” (ρ=0.143), “Getting ready for the state test takes a lot of time in our class.” ( ρ=0.103).  The report does not actually contain items that specifically mention “drill,”work their way methodically through the key concepts of literacy and mathematics,” or “taught to the test,” but I believe the reporters (and perhaps Gates officials) are referencing the test prep items with these phrases.

I sent links to the coverage and the Gates report to a half-dozen leading economists to ask if the claims mentioned above were supported by the findings.  The following reply from Jacob Vigdor, an economist at Duke, was fairly representative of what they said even if it was a bit more direct than most:

I looked carefully at the report and come to the same conclusion as you: these correlations are positive, not negative.  The NYT and LAT reports are both plainly inconsistent with what is written in the report.  A more accurate statement would be along the lines of “test preparation activities appear to be less important determinants of value added than [caring teachers, teacher control in the classroom, etc].”  But even this statement is subject to the caveat that pairwise correlations don’t definitively prove the importance of one factor over another.  Maybe the reporters are describing some other analysis that was not in the report (e.g., regression results that the investigators know about but do not appear in print), but even in that case they aren’t really getting the story right.  Even in that scenario, the best conclusion (given positive pairwise correlations and a hypothetically negative regression coefficient) would be that teachers who possess all these positive characteristics tend to emphasize test preparation as well.

Put another way, it’s alway good to have a caring teacher who is in control of the classroom, makes learning fun, and demands a lot of her students.  Among the teachers who share these characteristics, the best ones (in terms of value added) appear to also emphasize preparation for standardized tets.  I say “appear” because one would need a full-fledged multivariate regression analysis, and not pairwise correlations, to determine this definitively.

Another leading economist, who preferred not to be named, wrote: “I looked back over the report and I think you are absolutely right!”  I’m working on getting permission to quote others, but you get the idea.

In addition to confirming that a positive correlation for test prep items means that it contributes to value-added, not detracts from it, several of these leading economists emphasized the inappropriateness of comparing correlations to draw conclusions about whether test prep contributes to value-added any more or less than other teacher practices observed by students.  They noted that any such comparison would require a multivariate analysis and not just a series of pairwise correlations.  And they also noted that any causal claim about the relative effectiveness of test prep would require some effort to address the endogeneity of which teachers engage in more test prep.

As David Figlio, an economist at Northwestern University, put it:

You’re certainly correct here.  A positive pairwise correlation means that these behaviors are associated with higher performance on standardized tests, not lower performance.  The only way that it could be an accurate statement that test prep is causing worse outcomes would be if there was a negative coefficient on test prep in a head-to-head competition in a regression model — though even then, one would have to worry about endogeneity: maybe teachers with worse-performing students focus more on test prep, or maybe lower-performing students perceive test prep to be more oppressive (of course, this could go the other way as well.)  But that was not the purpose or intent of the report.  The report does not present this as a head-to-head comparison, but rather to take a first look at the correlates between practice measures and classroom performance.

There was no reason for this issue to have developed into the controversy that it has. The coverage contains obvious errors that should have been corrected quickly and clearly, just as Jason Felch is doing.   Tom Kane, Vicki Phillips, and other folks at Gates should have immediately issued a clarification as soon as they were alerted to the error, which was on Monday.

And while I did not know where the error occurred when I wrote the blog post on Monday, the indications now are that there was a miscommunication between the technical people who wrote the report and non-technical folks at Gates, like Vicki Phillips and the pr staff.  In other words, Sam Dillon can relax since the mistake appears to have originated within Gates (although Dillon’s subsequent defensiveness, name-calling, and failure to check with independent experts hardly bring credit to the profession of journalism).

The sooner Gates issues a public correction, the sooner we can move beyond this dispute over what is actually a sidebar in their report and focus instead on the enormously interesting project on which they’ve embarked to improve measures of teacher effectiveness.  An apology from Sam Dillon would be also nice but I’m not holding my breath.



“Academics” and the “Practical” Part III: The Daleks Are Coming!

December 15, 2010

(Guest post by Greg Forster)

This week Jay highlighted the fact that a study showing a positive correlation between test prep activity in the classroom and improved learning is being portrayed as having shown a negative correlation between test prep activity (“drill and kill”) and improved learning. At this point it’s not known (at least to me) where the error arose, and I don’t have anything to say about the question of who said what. But I think it illustrates how the whole subject of drillandkillaphobia needs to be revisited.

Lately I’ve explored in some depth how testing has come to be the focal point of the fight between the two great factions in the education space, the “liberal artists” and the “pragmatists.” Liberal artists have gradually come to invest all their hopes in standardized testing. Pragmatists have gradually come to invest all their fears in precisely the same thing. This is a mutually reinforcing circle – over time, the liberal artists increasingly think testing must be good because the pragmatists hate it, and the pragmatists increasingly think it must be bad because the liberal artists love it. So naturally the battle line over testing has become more and more absolute.

The problem, as I’ve tried to show, is that this dynamic causes each side to reject something that’s essential to good education. The liberal artists seek curricula (or standards that can only lead to curricula) that emphasize sterile head knowledge of facts to the exclusion of practical problem-solving. We don’t know how to test what Daniel Willingham calls “deep knowledge” of subject area content, and even if we did the tests would be almost infinitely vulnerable to manipulation if they were ever used for accountability purposes. But we do know how to create rigorous tests for head knowledge of facts, so the liberal artists define “subject area content” to mean simply head knowledge of facts.

The pragmatists do the opposite. And since I was pushing pretty hard last time to emphasize what I thought were the dysfunctions of the liberal artists, I’d like to balance the scales with something about drillandkillaphobia.

Merely mention the subject of testing and it seems that pragmatists instantly jump to the conclusion that you want schools to look like this:

Now the interesting thing is that these days, they will engage in the most hysterical drillandkillaphobia while all the time affirming that we need to keep standards high, content knowledge matters, etc. To some extent I think that must be intentionally tactical – P21 knows that its flavor of loosey-goosey crunchy granola doesn’t sell these days, since they’ve lost a lot of battles in the war of ideas. But I don’t think that explains all of it. Take another look at that Ken Robinson “Changing Paradigms” video, where he begins by saying that of course we don’t want to lower standards. Clearly he, at least, really means it. It seems completely obvious to him that there’s no contradiction between his attacks on testing and his affirmation of high standards.

And that’s the problem. It seems so obvious to him that he fails to even take the question seriously. (That was one of the key points in Willingham’s stimulating critique of Robinson’s video.)

If the liberal artists need to get over their testophilia, the pragmatists need to get over their drillandkillaphobia. I’m not aware of any hard evidence linking test prep to worse outcomes. Sure, lots of people are really convinced that it must be the case, but that’s hardly a solid ground for making policy. (The floor is open in the comments section if you have some hard evidence you want to share.) And it’s not like this is a new question. I’ll admit that I haven’t done an exhaustive review of the research (again, if you have, the floor is open) but Jay and I conducted a study a while back showing that attaching rewards and penalties to a test doesn’t change the results; that would seem to speak right to the heart of drillandkillaphobia. This new Gates Foundation study, finding a positive correlation between test prep and learning outcomes, would seem to be another piece of evidence against it.

People can’t learn what Willingham calls the “deep structure” of practical problems until they’ve learned what he calls the “surface structure.” You can’t get from the pool deck to the bottom of the pool without passing through the surface of the water; similarly, you can’t get to deep (i.e. practical) content knowledge without first getting shallow (i.e. factual) content knowledge. If you like, it’s “merely” or “sterile” head knowledge. But head knowledge it is and head knowledge it will remain, even after you add the “deep” part.

People learn head knowledge by memorization. And any kind of memorization will appear, to those who wish to stigmatize and delegitimize it, to be “merely” “rote” memorization. You can call it “regurtitation” when people know facts, and in a sense you’re right – but people do need to know facts and be able to summon up that knowledge as necessary, whether you call that “regurgitating” or not. And for everyone but the real genius students, gaining head knowledge of facts will involve some kind of “drill.”

As the pragmatists themselves never tire of reminding us, real learning is hard. Well, yes it is. You can’t learn if you don’t memorize stuff. Memorizing stuff is hard and unpleasant, and it’s a lot more so for some kids than for others. That’s the world; deal with it.

I fully admit that if you really want to learn you are never just memorizing. You must be trying to understand the facts you absorb – understand their significance and the connections between them. But while it must be more than memorizing, it is never less than memorizing. Of course, if it must be more then by definition it can never be less.

Take that great, perennial boogyman of rote memorization – historical dates. People whine, why does it matter in what year a certain event occurred? Well – why does it matter? If you stopped and seriously asked that question and sought out the answer, you might . . . well, you might learn something.

The great irony, of course, is that at the same time the pragmatists are pushing this new bout of drillandkillaphobia, they’re working hard to impose a federal-government controlled system of national testing – excuse me, a totally “voluntary” system of “common” “assessment” that has nothing to do with the federal government, nothing to see here, move along, pay no attention to the man behind the curtain. That really is an attempt to handcuff real learning and turn us all over to the benevolent dictatorship of soulless men in white coats who must be trustworthy because, after all, they’re scientists. And that, of course, was always the original sin of the Progressive movement, in its educational form as in all other forms. Handing over all power to a tiny priesthood is the very epitome of “democracy” as long as we’re careful to call the priests “scientists.” But now I’m broaching a whole new and much deeper subject, one that will require another post to handle with any justice.


False Claim on Drill & Kill

December 13, 2010

The Gates Foundation is funding a $45 million project to improve measures of teacher effectiveness.  As part of that project, researchers are collecting information from two standardized tests as well as surveys administered to students and classroom observations captured by video cameras in the classrooms.  It’s a big project.

The initial round of results were reported last week with information from the student survey and standardized tests.  In particular, the report described the relationship between classroom practices, as observed by students, and value-added on the standardized tests.

The New York Times reported on these findings Friday and repeated the following strong claim:

But now some 20 states are overhauling their evaluation systems, and many policymakers involved in those efforts have been asking the Gates Foundation for suggestions on what measures of teacher effectiveness to use, said Vicki L. Phillips, a director of education at the foundation.

One notable early finding, Ms. Phillips said, is that teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains than those who simply work their way methodically through the key concepts of literacy and mathematics. (emphasis added)

I looked through the report for evidence that supported this claim and could not find it.  Instead, the report actually shows a positive correlation between student reports of “test prep” and value added on standardized tests, not a negative correlation as the statement above suggests.  (See for example Appendix 1 on p. 34.)

The statement “We spend a lot of time in this class practicing for [the state test]” has a correlation of  0.195 with the value added math results.  That is about the same relationship as “My teacher asks questions to be sure we are following along when s/he is teaching,” which is 0.198.  And both are positive.

It’s true that the correlation for “Getting ready for [the state test] takes a lot of time in our class” is weaker (0.103) than other items, but it is still positive.  That just means that test prep may contribute less to value added than other practices, but it does not support the claim that  “teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains…”

In fact, on page 24, the report clearly says that the relationship between test prep and value-added on standardized tests is weaker than other observed practices, but does not claim that the relationship is negative:

The five questions with the strongest pair-wise correlation with teacher value-added were: “Students in this class treat the teacher with respect.” (ρ=0.317), “My classmates behave the way my teacher wants them to.”(ρ=0.286), “Our class stays busy and doesn’t waste time.” (ρ=0.284), “In this class, we learn a lot almost every day.”(ρ=0.273), “In this class, we learn to correct our mistakes.” (ρ=0.264) These questions were part of the “control” and “challenge” indices. We also asked students about the amount of test preparation they did in the class. Ironically, reported test preparation was among the weakest predictors of gains on the state tests: “We spend a lot of time in this class practicing for the state test.” (ρ=0.195), “I have learned a lot this year about the state test.” (ρ=0.143), “Getting ready for the state test takes a lot of time in our class.” ( ρ=0.103)

I don’t know whether something got lost in the translation between the researchers and Gates education chief, Vicki Phillips, or between her and Sam Dillon at the New York Times, but the article contains a false claim that needs to be corrected before it is used to push changes in education policy and practice.

UPDATE —

The LA Times coverage of the report contains a similar misinterpretation: “But the study found that teachers whose students said they “taught to the test” were, on average, lower performers on value-added measures than their peers, not higher.”

Try this thought experiment with another observed practice to illustrate my point about how the results are being mis-reported…  The correlation between student observations that “My teacher seems to know if something is bothering me” and value added was .153, which was less than the .195 correlation for “We spend a lot of time in this class practicing for [the state test].”  According to the interpretation in the NYT and LA Times, it would be correct to say “teachers who care about student problems tend to have lower value-added learning gains than those who spend a lot of time on test prep.”

Of course, that’s not true.  Teachers caring about what is bothering students is positively associated with value added just as test prep is.  It is just that teachers caring is a little less strongly related than test prep.  Caring does not have a negative effect just because the correlation is lower than other observed behaviors.

(edited for typos)


Klein’s Lessons

December 6, 2010

*** Joel Klein tells us in the pages of the WSJ what he learned as Chancellor of NYC schools.  Here’s a highlight:

First, it is wrong to assert that students’ poverty and family circumstances severely limit their educational potential. It’s now proven that a child who does poorly with one teacher could have done very well with another. Take Harlem Success Academy, a charter school with all minority, mostly high-poverty students admitted by lottery. It performs as well as our gifted and talented schools that admit kids based solely on demanding tests. We also have many new small high schools that replaced large failing ones, and are now getting outsized results for poor children.

Second, traditional proposals for improving education—more money, better curriculum, smaller classes, etc.—aren’t going to get the job done. Public education is a service-delivery challenge, and it must be operated as such.

Klein raises an excellent point.  Diane Ravitch, Sol Stern, and others who claim that they have grown frustrated with choice and other incentive-based reforms because they haven’t yet produced the miracles they expected ought to be 1,000 times more frustrated with the failure of more money, higher teacher certification requirements, curricular and pedagogical reform, etc…  We’ve tried those kinds of reforms more than 1,000 times more on a much grander scale and yet we still wait for the miracles.

To judge the effectiveness of reform strategies we can’t use miraculous improvement as the standard.  And we certainly need more fine-grained analyses than looking at whether cities or states that have tried something have improved.  And finally, we can work on various types of reforms simultaneously, so pitting incentives versus instruction is a false conflict that serves only to inflate the ego-starved reformer rather than the cause of reform.


“Academics” and the “Practical” Part II: Neither Just Skills nor Just Facts

December 1, 2010

(Guest post by Greg Forster)

Picking up (after a month of constant travel) on my effort to build a unified field theorem of education reform, I want to respond to a strong challenge Jay issued to my use of the word “skills” in my original post.

I used the term “basic skills” to refer to the three Rs, without even thinking about it. I didn’t even stop to define it; after Jay challenged me I thought he had misunderstood what I meant by that term, so I went back and posted a clarification. But it turned out Jay had understood me perfectly well. He just wanted to challenge that use of the term.

I think this is likely to be a crucial issue in my effort to reconcile the legitimate interests of “liberal artists” with the legitimate interests of “pragmatists,” so it’s worth pausing to hash it out.

Jay pointed me to this article by Daniel Willingham: “Critical Thinking: Why Is It So Hard to Teach?” I haven’t had nearly enough time to digest it fully, I’m afraid, but I’ve digested enough of it to offer what I think is a useful next step in thinking about this issue.

First off: In a discussion of the controversy between teaching the three Rs (and all the other things liberal artists want) and teaching critical thinking (and all the other things pragmatists want), why did Jay challenge my conception of what it means to teach the three Rs and then, to back up his challenge, point me to an article on critical thinking? Why not an article on the three Rs?

Because this is, really, an article on the three Rs, and on the whole liberal arts agenda more generally, disguised as an article on critical thinking. Presumably that’s how he got it published in American Educator – an organ of the AFT! It’s the educational equivalent of the Sokal hoax. Willingham has a bunch of pragmatists – teacher union pragmatists, no less – publishing liberal artist propaganda. It’s a brilliant practical joke.

…or so I thought when I first looked at it.

But the more I look at it, the more I think the joke is as much on us liberal artists as it is on the pragmatists.

Willingham’s thesis in a nutshell:

First, critical thinking (as well as scientific thinking and other domain-based thinking) is not a skill. There is not a set of critical thinking skills that can be acquired and deployed regardless of context. Second, there are metacognitive strategies that, once learned, make critical thinking more likely. Third, the ability to think critically(to actually do what the metacognitive strategies call for) depends on domain knowledge and practice.

The idea that critical thinking isn’t a skill is the real core of the article:

Critical thinking does not have certain characteristics normally associated with skills—in particular, being able to use that skill at any time. If I told you that I learned to read music, for example, you would expect, correctly, that I could use my new skill (i.e., read music) whenever I wanted. But critical thinking is very different. As we saw in the discussion of conditional probabilities, people can engage in some types of critical thinking without training, but even with extensive training, they will sometimes fail to think critically. This understanding that critical thinking is not a skill is vital. It tells us that teaching students to think critically probably lies in small part in showing them new ways of thinking, and in large part in enabling them to deploy the right type of thinking at the right time.

Rather, critical thinking is what emerges when you do a good job of teaching content. That’s because learning “to deploy the right type of thinking at the right time” only happens when you learn a specific field. You don’t learn the scientific method by studying the scientific method (which is what is implied by calling it a “skill”). You can learn about the scientific method that way; that is, you can acquire “head knowledge” of facts about it, facts that could be regurgitated on a test. But it doesn’t make you any better at actually using the scientific method. Studying biology, on the other hand, will make you better at the scientific method because you’re actually using it. It will even make you better at using the scienfic method in all other disciplines, even the ones you haven’t studied. That’s because in your study of biology you’re learning 1) knowledge of the “deep structure” of problems in that field, and 2) “contextual cues” in that field that signal you when to do what. That knowledge, and not “the scientific method” learned as a skill in its own right, is what helps you figure out “deep structure” and “contextual cues” in other fields.

So the pragmatists, who want to focus on “critical thinking” as such, have it all wrong, and the liberal artists, who want to focus on teaching content, have it all right. Right?

Wrong.

Throughout Willingham’s analysis there is an emphasis on how critical thinking emerges from learning these content-specific disciplines in practice. You learn “deep structure” and “contextual cues,” not by studying deep structure and contextual cues as such, but by learning specific disciplines like biology. However, you learn deep structure and contextual cues in biology, not by reading books about biology, but by doing biology. You conduct experiments, you do field research, etc. You go out and solve problems and create knowledge.

Look again at one particular phrase in my description of Willingham above: studying the scientific method as such wouldn’t make you any better at using the scientific method, but it would give you facts about it – the kind of thing you could regurgitate on a test.

And there’s our problem. We liberal artists really do have a strong tendency to reduce content knowledge to “head knowledge” of facts. If you can pass a test on a subject, you know the subject. But – and here we are hoist on Willingham’s petard – that is not the kind of content-specific knowledge that leads to good critical thinking.

Obviously we need to have head knowledge. Students need to learn facts. That’s vital. In particular, for all the reasons outlined in my original post, I think the emphasis on standardized testing emerged for good reasons, and standardized testing needs to remain an important part of our educational landscape.

And I’m not giving up my position that the pragmatists, in their zeal to equip students with critical thinking, creativity, the entrepreneurial mindset, etc. have historically sought these qualities at the expense of, rather than in addition to, content knowledge – and that this has historically had devastating effects.

But I’m also sounding a red alert that we liberal artists have gone just as far wrong in allowing our zeal for accountability – which in practice has come to mean “testing” for too many of us – to drive us into a reductionistic approach to what content knowledge really consists of.

I wonder if it would help to go back to Aristotle’s concept of “intellectual virtues.” He classified the goal of education as imparting not skills or facts, but virtues. And alongside the “moral virtues” he put “intellectual virtues.” Indeed, he thought the two were not just equally important but interdependent; you couldn’t have one without the other.

He warned that a “virtue,” whether moral or intellectual, cannot be reduced to either just a personal characteristic we possess or a thing that we do. If a virtue is just a characteristic, then we’re “virtuous” while asleep or in a coma; if a virtue is just a thing that we do, then our “virtue” depends as much on circumstances outside us as on our character. Rather, virtue must be something that is both active and intrinsic.

Aristotle solved this problem by proposing that virtue is a habit. To possess a virtue means to be in the habit of doing the right things at the right times.

Now, I suppose (stretching just a little bit) that the reductionistic tendency of the pragmatists is to pursue their goals – critical thinking and so forth – as merely something they want students to do. They seek the activity but not the intrinsicness. And the reductionistic tendency of the liberal artists is to pursue their goals as merely something they want students to have inside them; they seek the intrinsicness but not the activity. It might help us to start thinking of learning as the imparting of good habits – to intrinsically possess the quality of being prone to do the right things.

But whether Aristotle helps or not, it seems to me that a recovery of good education must be neither a turn away from the practical toward the academic, nor a turn away from the academic toward the practical, but an acknowledgement that, by separating the two, we have really lost both.