Build New, Don’t Reform Old

August 2, 2011

Image result for gates foundation headquarters

When I wrote my two part critique of the Gates Foundation strategy, one of our frequent comment-writers, GGW, asked: “What would you do if asked by Gates how to better donate his (and Warren Buffett’s) billions?”

Here is a brief answer to that question: Philanthropists with billions of dollars to devote to education reform should build new institutions and stop trying to fix old ones.

In general, existing institutions don’t want to be fixed.  There are reasons why current public schools operate as they do and the people who benefit from that will resist any effort to change it.  Those who benefit from status quo arrangements also tend to be better positioned than reformers to repel attempts by outsiders to make significant changes.  The history of education reform is littered with failed efforts by philanthropists.

Instead, private donors have had much better success addressing problems by building new institutions.  And competition from newly built institutions can have a greater positive impact on existing institutions than trying to reform them directly.

Let’s consider one of the greatest accomplishments in American education philanthropy.  In the late 19th century, America’s leading universities (Harvard, Yale, Princeton) were badly in need of reform.  They were still operated primarily as religious seminaries and not as modern, scientific institutions.  Rather than trying to reform them directly, major philanthropists built new universities modeled after German scientific institutions.  John D. Rockefeller and Marshall Field helped found the University of Chicago.  Leland Stanford built Stanford University.  A group of private donors built Johns Hopkins.  Cornelius Vanderbilt founded Vanderbilt.  All of these universities imitated German universities with their emphasis on the scientific method and research and were enormously successful at it.  Eventually Harvard, Yale, and Princeton recognized the competitive threat from these German-modeled upstarts and made their own transition from a seminary-focus to a scientific focus.

The reform of the U.S. higher education system did not come from a government mandate or “incentives.”  It did not happen by philanthropists giving money directly to the leading universities of the time to convince them to change their ways.  It happened by philanthropists building new institutions to compete with the old ones.

The same could be done for K-12 education.  Matt Ladner has written a series of posts on “The Way of the Future.”  He, along with Terry Moe, Clay Christensen, Paul Peterson, and others, envision large numbers of  hybrid virtual schools offering higher quality customized education at dramatically lower costs.  Students would attend school buildings, but the bulk of their instruction would be delivered by interactive software.  The school would need significantly fewer staff, who would concentrate mostly on assisting students with the technology and managing behavior.

Obviously, this kind of school would not be good for everybody.  But it could appeal to large numbers of students and be offered at such a low cost that it could be affordable even to low-income families without needing public subsidy or adoption by the public school system.

Gates or someone else with billions to devote to education could build a national chain of these virtual hybrid schools to compete with existing public and private schools.  It’s true that Gates is already investing in the development and refinement of the virtual hybrid school model, but a complete commitment to building new rather than reforming old would give him the potential to do what Rockefeller, Stanford, and others did to higher education.  Virtual hybrid schools could be the disruptive technology, as Christensen calls it, to produce real reform in education.

Another benefit of the “building new” strategy for philanthropists is that it avoids the Emperor’s New Clothes problem, where philanthropists are encouraged to pursue flawed strategies to reform existing institutions because everyone is afraid to criticize the wealthy donor from whose largess they benefit.  With the “build new” strategy there is ultimately a market test of the wisdom of the strategy.  If the new institutions are not better, people won’t choose them.  If the University of Chicago had been a flawed model, it wouldn’t have attracted enrollment and would have failed to apply competitive pressure to Harvard, Yale, and Princeton.  Similarly, if the virtual hybrid school is a bad model, then it won’t attract students and compete with existing public and private schools.

Edison Schools is an example of a “build new” strategy that failed the market test.  They failed to develop technologies or other efficiencies to bring down the costs of operating private schools.  And their revised strategy of operating public schools under contract with public school districts was flawed by an underestimation of the political resistance they would face and their inability to control costs or quality within the public system.

But we also have successful examples of the “build new” strategy adopted by philanthropists.  In addition to the string of scientific universities built in the latter half of the 19th century, we also have the example of Andrew Carnegie and public libraries.  Carnegie helped promote literacy and cultural knowledge by supporting the construction of hundreds of new libraries around the country.  He didn’t try to reform existing book-sellers, he just built new.  Another example (outside of education) can be seen in John D. Rockefeller’s role in the development of a national park system.  Rockefeller privately acquired large chunks of what are now the Acadia, Grand Teton, Great Smoky Mountains and Yellowstone national parks.  Rockefeller didn’t try to reform the operations of the existing Interior Department.  Instead, he effectively privately built nature reserves and then donated them to the U.S. to become national parks.

Of course, this “build new” strategy has limited potential for smaller-scale philanthropy.  But for the very wealthy, like Gates, the path to making a significant and lasting difference is to build new rather than reform old.  The lasting benefits of what Rockefeller did in higher education and national parks and Carnegie did with libraries are still noticeable today.  If Gates and others with billions to devote to education continue to focus on reforming the old rather than building new, I fear their efforts will soon be forgotten after the Emperor’s New Clothes adulation fades when they stop having large sums to give.

22 Comments | economics, National Standards | Tagged: , | Permalink
Posted by Jay P. Greene

Gates Foundation Follies (Part 2)

July 26, 2011

Image result for gates foundation headquarters

A sketch of the $500 million new Gates Foundation headquarters

In Part 1 of this post, I described how the Gates Foundation came to recognize the importance of using political influence to reform the education system rather than focusing on reforming one school at a time in the hopes that school systems would see and replicate successful models.  No private philanthropist has enough money to buy and sustain widespread adoption of an effective approach and the public school system has little incentive to identify and spread effective approaches on their own.

Faced with the unwillingness of the public school system to reproduce successful models (assuming that Gates could even offer one), the Foundation was left with two solutions to encourage innovation: 1) identify the best practices themselves and impose them from the top down, or 2) encourage choice and competition so that schools would have the proper incentive to identify, imitate, and properly implement effective approaches.

The Gates Foundation made the wrong choice.  Their top-down strategy cannot work for the following reasons:

1) Education does not lend itself to a single “best” approach, so the Gates effort to use science to discover best practices is unable to yield much productive fruit;

As I’ve explained before, there are many different “best” techniques for different kinds of teachers with different kinds of students in different situations with different available resources.  There are some practices that are universally beneficial in education, but they tend to be pretty obvious and are already well known (e.g. it is bad to beat kids, it is better when teachers know the material they are teaching, it is helpful to break down ideas into their essential components, etc…).

The difficulty of discovering universally beneficial  practices that are not already well-known, especially with the blunt tools available to researchers probably helps explain why the Measuring Effective Teachers (MET) project, on which the Gates Foundation is spending $335 million has yet to produce any meaningful results despite entering its third year of operation.

2) As a result, the Gates folks have mostly been falsely invoking science to advance practices and policies they prefer for which they have no scientific support;

Despite having nothing to show for the $335 million they are spending on MET, the Gates folks nevertheless claim that it “proves” the harmfulness of teachers engaging in “drill and kill.” The fact that the research showed no such thing did not deter them from telling the NY Times and LA Times that it did.  Even when I pointed out the error, the Gates folks refused to issue a correction (although the LA Times ran one on their own).

Similarly, the Gates-orchestrated effort to push national standards, curricular materials, and assessments is advancing without any scientific evidence of the desirability of these approaches.  Gathering a group of Checker Finn’s friends (er, I mean, “a panel of experts”) to attest that the Common Core standards are better is not science.  It is the false invocation of science to manipulate people into compliance with their agenda.

3) Attempting to impose particular practices on the nation’s education system is generating more political resistance than even the Gates Foundation can overcome, despite their focus on political influence and their devotion of significant resources to that effort;

Opponents of centralized control of education have begun to mobilize against the Gates-orchestrated effort to establish national standards, curricular materials, and assessments.  But the bulk of the political resistance to the Gates strategy will come from the teacher unions.  They don’t want anyone to infringe on their autonomy or place their interests in jeopardy with a nationalized accountability system.  They may play along with Gates for a while and take their money, but when push comes to shove the unions can only tolerate one dictator in education — the unions.  Of course, those of us who don’t want anyone centrally-controlling the nation’s education system will oppose both Gates and the teacher unions.

We already have a taste of the kind of resistance teacher unions will put up against the Gates nationalization effort in the slogans emanating from Diane Ravitch and Valerie Strauss’ Twitter feed, supported by their Army of Angry Teachers.  Falsely claiming that MET proved that drill and kill is harmful did not mollify these folks at all.

The teacher unions derive far more power and money from the status quo than Gates can ever offer them, unless of course Gates builds a nationalized system and cedes control to the unions, which is not part of the Gates plan.  Nothing in the Gates strategy weakens the unions and would force them to make significant concessions, so in the end the unions will either hijack the Gates strategy for their own benefit or block it.  Even Gates does not have the resources to beat the unions without first diminishing their power.

4) The scale of the political effort required by the Gates strategy of imposing “best” practices is forcing Gates to expand its staffing to levels where it is being paralyzed by its own administrative bloat; 

Over the last decade the Gates Foundation has roughly doubled its assets but increased its staffing by about 10-fold.  The Foundation is now huge, which is part of why it needs the Education Pentagon pictured above to house everyone.  The Foundation has gotten huge because it is trying to buy political influence as it buys people.  Gates has been snapping up or funding just about every advocacy group, researcher, or education journalist they can find.  Getting all of these people on board for a nationalized education system (or at least mute their dissent) involves paying an enormous number of people and organizations.

Gates can buy a lot of folks, but they can’t buy everyone and they can’t keep the folks they do pay in line for very long.  It’s like herding cats. (I should note that I’ve received Gates Funding in the past).

And the sheer size of their staff and funded allies along with the focus on controlling the political message is so overwhelming that it is significantly hindering their ability to do anything.  People inside the organization have told me that they are suffering from a bureaucratic gridlock with endless meetings, conference calls, and chains of approvals.  Notice that Gates is paying a ton of researchers and yet virtually no research is coming out.  Very curious.

5) The false invocation of science as a political tool to advance policies and practices not actually supported by scientific evidence is producing intellectual corruption among the staff and researchers associated with Gates, which will undermine their long-term credibility and influence.

As noted above, the need to advance a particular political message has led Gates to mischaracterize their own research (for example, claiming that MET proves that drill and kill is harmful when the research does not show that).  But the intellectual corruption extends much farther.  I had a highly respected and accomplished researcher employed by Gates tell me that Vicki Phillips’ mischaracterization of the MET results was not so far off because there isn’t a big difference between a low correlation and a negative one.  He also defended comparing the magnitude of a series of pair-wise correlations to determine the relative influence of different variables.  To hear someone who knows better twist the truth to avoid contradicting the education boss at Gates was just sad.

Unfortunately, too many advocates, researchers, and others are being similarly corrupted.  In most cases the Gates folks don’t have to exert any explicit pressure on people to keep them in line; they just anticipate what they think would serve the Gates strategy.  But I am aware of at least one case in which a researcher’s findings were at odds with the desired outcome and that person suffered for it.

I’ve heard another story from someone involved in the MET project that the delay in releasing any results from the analyses of classroom videos even as the project enters its third year is explained by their inability to find any meaningful results.  Perhaps another year of data will make something turn up that they can finally tout for their $335 million investment.  The fact that the initial MET report with basically no useful findings was released on a Friday just before Christmas suggests that the Gates folks are working hard to shape their message.

The national standards, curriculum, and testing campaign is rife with intellectual corruption.  For example, people are twisting themselves into knots to explain how the effort is purely voluntary on the part of states when it is manifestly not, given federal financial “incentives,” offers of selective exemptions to NCLB requirements for states that comply, and the threat of future mandates.  There is so much spin around Gates that it makes one dizzy.

—————————————————————————————–

Let me be clear, most of the folks affiliated with Gates are good and smart people.  The problem is that when your reform strategy requires a top-down approach, these good and smart people are put under a lot of stress to have a unified vision of the “best” that will be imposed from the top.  And whenever an organization starts sprinkling millions of dollars on researchers and advocacy groups unaccustomed to that kind of money, there are temptations that are hard for the most virtuous to resist.

But the good and smart people at Gates can stop the counter-productive strategy that the Foundation is pursuing.  The Foundation changed course once before and it can do it again.

————————————————————————————–

UPDATE — For my suggestions of what the Gates Foundation could do instead, see this post.

31 Comments | instructional reform, politics | Tagged: | Permalink
Posted by Jay P. Greene

Gates Foundation Follies (Part 1)

July 25, 2011

Image result for gates foundation headquarters

A sketch of the $500 million new Gates Foundation headquarters

Jason Riley’s interview with Bill Gates in the Wall Street Journal was not as great as Riley’s interview with me last week (shameless plug for my new mini-book), but it was still very illuminating.  In particular, the Gates interview confirmed two things about the Foundation’s education efforts: 1) they’ve realized that the focus of their efforts has to be on the political control of schools and 2) they are uninterested in using that political influence to advance market forces in education. Instead, the basic strategy of the Gates Foundation is to use science (or, more accurately, the appearance of science) to identify the “best” educational practices and then use political influence to create a system of national standards, curricular materials, and testing to impose those “best practices” on schools nationwide.

The Gates Foundation came to understand the necessity of political influence over schools with the failure of their previous small schools strategy.  Under that strategy they tried to achieve reform by paying school districts to break-up larger high schools into smaller ones.  The problem with that strategy is that even the Gates Foundation does not have nearly enough money to buy systemic reform one school at a time.

School districts currently spend over $600 billion per year and the Gates Foundation only has $34 billion in total assets.  With the practice of spending only about 5% of assets each year and given the large (and effective) efforts the Foundation makes in developing country health-care, Gates only spends a couple hundred million dollars on education reform each year. Given the small share of total education spending Gates could offer, most public districts refused to entertain the Gates strategy of smaller schools, others took the money but failed to implement it properly, and others reversef the reform once the Gates subsidies ended.

The way I described the situation in my chapter “Buckets into the Sea” in the 2005 book, With the Best of Intentions, edited by Rick Hess is:

Philanthropists simply don’t have enough resource to reshape the education system on their own; all their giving put together amounts to only a tiny fraction of total education spending, so their dollars alone can’t make a significant difference.  In order to make a real difference, philanthropists must support programs that redirect how future public education dollars are spent.

And in 2008 I repeated this claim, saying: “total private giving to public education is a tiny portion of total spending on schools.  All giving, from the bake sale to the Gates Foundation, makes up less than one-third of 1% of total spending.  It’s basically rounding error.”

I don’t know whether the Gates Foundation was influence by my writing or whether they arrived at the same conclusions independently, but they are now articulating those same conclusions, often with the same exact words:

“It’s worth remembering that $600 billion a year is spent by various government entities on education, and all the philanthropy that’s ever been spent on this space is not going to add up to $10 billion. So it’s truly a rounding error.”

This understanding of just how little influence seemingly large donations can have has led the foundation to rethink its focus in recent years. Instead of trying to buy systemic reform with school-level investments, a new goal is to leverage private money in a way that redirects how public education dollars are spent.

While the focus of the Gates Foundation on influencing education policy is sensible, the particular political approach they have chosen is doomed to fail and attempting it is likely to be counter-productive.  In Part 2 of this post I will explain how the new strategy Gates has decided to pursue is flawed.

To give you a taste of what is coming in Part 2, the arguments can be summarized as: 1) Education does not lend itself to a single “best” approach, so the Gates effort to use science to discover best practices is unable to yield much productive fruit; 2) As a result, the Gates folks have mostly been falsely invoking science to advance practices and policies they prefer for which they have no scientific support; 3) Attempting to impose particular practices on the nation’s education system is generating more political resistance than even the Gates Foundation can overcome, despite their focus on political influence and their devotion of significant resources to that effort; 4) The scale of the political effort required by the Gates strategy of imposing “best” practices is forcing Gates to expand its staffing to levels where it is being paralyzed by its own administrative bloat; and 5) The false invocation of science as a political tool to advance policies and practices not actually supported by scientific evidence is producing intellectual corruption among the staff and researchers associated with Gates, which will undermine their long-term credibility and influence.

Tune in for Part 2.

————————————————————————————–

UPDATE — For my suggestions of what the Gates Foundation could do instead, see this post.

6 Comments | instructional reform, politics | Tagged: | Permalink
Posted by Jay P. Greene

The Fordham Report is Here. Time to Party!

April 19, 2011

The Fordham report on renewal of ESEA has been released and it is time to party!

Following the rules of our Fordham report drinking game you will have to consume 7 shots of your choice; one for each time “tight-loose” is used in the report.  33 times you will need to consume whatever the Gates Foundation and U.S. Department of Ed mandate while declaring “I do this of my own free will;” one for each usage of “Common Core” in the report.  You need to shotgun a Pabst Blue Ribbon for the 1 usage of “race to the bottom” in the report and consume 8 Milwaukee’s Best for the 8 times “Race to the Top” is used.  That’s 42 total “consumptions.”

I whiffed on predicting the usage of “smart-[blank].”  I’m sorry to say that there was nothing very smart in the report.  I also entirely failed to expect the repeated usage of the phrase, “reform realism.”  It has alliteration!  What could be more persuasive than that?  I guess that is why it appears 21 times in the report.

Greg did accurately anticipate a slew of hemisphere fallacies, where they compromise between the view that the world is a sphere and the world is flat by saying that the world is a hemisphere.  The particular manifestation of the hemisphere fallacy in this report is that they repeatedly frame the debate as saying that some people think that the federal government should mandate something (standards, cut scores, etc…) and some people think that the federal government should mandate nothing in exchange for the resources it provides.  Fordham takes the middle ground of saying that the feds should mandate standards, cut scores, etc… or allow states to prove to a panel of experts that their alternative approach is at least as good.

Where to begin?  First, in practice the Fordham approach is equivalent to the feds mandating standards, cut scores, etc… If I told you that you had to eat the food the government provides or prove that your choices were equally nutritious, most people would end up just eating whatever the government provided.  The burden of proving the merit of your alternative choices would effectively compel you to comply with the mandate.

Second, if there is one thing we do not need in education policy, it is more committees of so-called experts.  Fordham proposes a bizarre procedure by which the expert panelists could be selected.  States would choose two members, the secretary of education would propose two more, and those four would choose an additional three panelists.  And if that is not convoluted enough, the panels would need 5 votes to decide anything.  This doesn’t sound like a committee of experts.  This sounds like politics by other means.  And given how complicated and bizarre this procedure is, it is even more likely that states would simply comply with the mandate, as suggested above.

Third, as is usual with hemisphere fallacies, Fordham frames the alternative “extremes” as caricatures so that their middle position seems like the only sensible alternative.  It isn’t.  I support a limited role of the federal government in education to facilitate the education of students who are significantly more expensive to educate, such as disabled students, English language learners, and students from very disadvantaged backgrounds.  Only the federal government can ensure this type of “redistributive” policy in education because if localities attempted to serve more expensive students they would attract those expensive students while driving away their tax base.  As Paul Peterson described in his classic book, The Price of Federalism, this is the only appropriate role of the federal government in education.  So, the federal government mandates that schools serve these categories of students while also providing additional resources to facilitate that the services will be provided.  This redistributive effort describes the bulk of what the federal government has done (and should do) in education.

If we are concerned that local schools are failing to serve these categories of students adequately we can address (and have imperfectly addressed) that through legal remedies.  Families, at least in special ed, can go to the courts if their schools fail to provide an appropriate education with federal funds.  We could expand that model to the other categories of federal involvement, but I think that approach is unwise.  Instead, I would favor providing the federal funds directly to students in these redistributive categories so that they would have economic leverage over schools to ensure the provision of appropriate services.  If schools fail to address student needs, they should be able to take those federal funds to another school, public or private.

The other phrase that I should have included in our drinking game is “college and career readiness.”  That concept is referenced 44 times in the new Fordham report.  It is the criterion by which expert panels need to judge standards, cut scores, etc… It is the goal of the entire Fordham approach (and remarkably in sync with the Gates Foundation in using a phrase dozens of times that was virtually unheard of a decade ago).

The only problem is that I have no idea what “college and career readiness” means.  The Fordham folks have no idea what that phrase means.  No one knows what college and career ready means.  It has no clear, technical, objective definition.  It is yet another political slogan substituting for an idea with actual substance, sort of like “reform realism” or “tight-loose.”

And yet this empty slogan is the entire purpose of the nationalization project on which Fordham-Gates-AFT-U.S. Dept of Ed are embarked.  Only in the D.C. bubble of  power-hungry analysts who provide no actual analysis could we launch a radical transformation of our education system with little more than a series of empty slogans.  It’s enough to make you drink.  Er, I mean consume.

(edited for clarity)

10 Comments | instructional reform, school accountability | Tagged: , , , | Permalink
Posted by Jay P. Greene

The Educationist View of Math Education

January 23, 2011

(Guest post by Barry Garelick)

In Jay Greene’s recent blog post, “The Dead End of Scientific Progressivism,” he points out that Vicki Phillips, head of education at the Gates Foundation misread her Foundation’s own report.  Jay’s point was that Vicki continued to see what she and others wanted to see: “‘Teaching to the test makes your students do worse on the tests.’ Science had produced its answer — teachers should stop teaching to the test, stop drill and kill, and stop test prep (which the Gates officials and reporters used as interchangeable terms).”

I was intrigued by the education establishment’s long-held view as Jay paraphrased it.  This view has become one of the “enduring truths” of education and I have heard it expressed in the various classes I have been taking in education school the last few years.  (I plan to teach high school math when I retire later this year).  In terms of math education, ed school professors distinguish between “exercises” and “problems”.  “Exercises” are what students do when applying algorithms or routines they know and can apply even to word problems. Problem solving, which is preferred, occurs when students are not able to apply a mechanical, memorized response, but rather have to apply prior knowledge to solve a non-routine problem.  Moreover, we future teachers are told that students’ difficulty in solving problems in new contexts is evidence that the use of “mere exercises” or “procedures” is ineffective and they are overused in classrooms.  One teacher summed up this philosophy with the following questions: “What happens when students are placed in a totally unfamiliar situation that requires a more complex solution? Do they know how to generate a procedure?  How do we teach students to apply mathematical thinking in creative ways to solve complex, novel problems? What happens when we get off the ‘script’?”

As someone who learned math largely though mere exercises and who now creatively applies math at work, I have to question this thinking. I believe that students’ difficulty in solving new problems is more likely to be because 1) applying prior knowledge to new or non-routine problems is hard for everyone, and 2) it is even harder for students who may lack the requisite knowledge and/or mastery of skills—not because they possess such knowledge and mastery.   So while the educationists distinguish between “exercises” and “problems”, the view refuses to distinguish between novices and experts.

Daniel Willingham, a cognitive scientist who teaches at the University of Virginia, finds the distinction between novice and expert to be quite important.  He maintains that it takes time and effort for knowledge to accumulate to the point that connections between learned material and new and difficult problems can be made.  Willingham refers to the difficulty that novices have with thinking critically as “inflexible thinking.”  In fact, he characterizes such difficulty as perfectly normal and to be expected among students.  Willingham argues that understanding the deep structures of a discipline such as mathematics is an important goal of education, “but if students fall short of this, it certainly doesn’t mean that they have acquired mere rote knowledge and are little better than parrots.” Rather, they are making the small steps necessary to develop better mathematical thinking. Simply put, no one leaps directly from novice to expert.

I was therefore extremely interested to see the sample problem of the Balanced Assessment in Mathematics (BAM) in Appendix 1 of the Gates Foundation’s preliminary report. The BAM is to be used to assess teacher effectiveness and according to the Gates Foundation’s preliminary report, “In comparison to many other assessments, BAM is considered to be more cognitively demanding and measures higher order reasoning skills using question formats that are quite different from those in most state mathematics achievement tests. There is also some evidence that BAM is more instructionally sensitive to the effects of reform-oriented instruction than a more traditional test (ITBS).”

The sample problem is reproduced below and is from the 8th grade mathematics assessment:

The diagram makes it clear how one is to count the number of tiles needed, so the first question is relatively easy.  Question 2 requires more thought, and the student must be able to extend the counting algorithm for the 4ft pond in question 1 to ponds of other sizes; i.e., they must understand that 4 is added to the product of the number of one foot long squares that fit on one side of the pond times 4, or 4n + 4 where n is the length of one side of the square pond in feet.

There are four additional questions that become increasingly difficult:  “How many paving stones are needed to surround a fish pond that is 20 feet by 20 feet?”, “Chris has 48 paving stones.  Find the size of the largest square pond the paving stones can surround.”  “The garden center sells many different sizes of square fish ponds. Write down a rule that will help Chris figure out how many paving stones are needed to surround square  ponds of different sizes.”  “The garden center decides to sell rectangular ponds.  Find a rule that will help Chris figure out how many paving stones are needed to surround rectangular ponds of different sizes.”

This set of questions is fairly challenging to beginning algebra students. (It is even more challenging if they have had no algebra at all, but since this is a problem for 8th grade students, I am assuming that they have had some experience with algebraic expressions and equations. This is not always a safe assumption but that’s a topic for another article.) The sample problem is illustrative of the type of problem that the education community deems coach-proof since it appears that memorization of problem solving techniques and “drill and kill” exercises will not work here.  But in fact, practice with some exercises would help students in tackling such a problem—specifically, having students express in mathematical terms certain situations described in English.  For example, “Three more than two times what John’s age will be in five years” (3 + 2(x + 5) ).  These types of exercises are frequently deemed by the education establishment to be inauthentic and irrelevant to the deeper underlying concepts of math unlike “reform oriented instruction” which purportedly provides such deep understanding through so-called authentic problems and a minimum of “exercises”.

Interestingly, the TIMSS exam—an international exam given every three years—also contains questions of this sort, as well as more straightforward problems.  For comparison’s sake, I looked at TIMSS eighth grade questions in algebra (found at http://nces.ed.gov/timss/pdf/TIMSS8_Math_ConceptsItems_2.pdf ) and found a similar type of problem:

This problem requires students to understand and ultimately express the relationship between the number of smaller squares on each side of the larger square, and the number of triangles contained in the square.  Question C, in fact, requires the student to be able to express the relationship mathematically in order to calculate the number of triangles.  The TIMSS report in which this appears provides some interesting data related to this question; namely the percent of students taking the test in each country that obtained the correct answer:

The top five scoring countries for this question ranged from 44 to 49% correct.  For the more straightforward problems the top five scores tend to be in the 70 and 80 percent range.   The US students obtained 19 percent correct for this problem; on more straightforward problems, the US scores in the 50 and 60 percent range. Thus, for all students, regardless of country, non-routine problems prove to be difficult.  Of interest, however, is that the five top scoring countries for this particular problem are Asian,  frequently criticized for using drilling and “inauthentic” exercises which they maintain do not properly prepare students for solving non-routine problems.

If Vicki Phillips’ statement about teaching to the test is any indication, however, the educational establishment will see what they want to see.  They will likely proclaim that the higher scores obtained by Asian countries on non-routine problems serves as evidence that the Asian countries use “reform-oriented instruction”.  Either that, or they’ll shrug their shoulders and say “It’s the culture; what can you do?”  (See http://www.educationnews.org/commentaries/104502.html )

In any event, whether the education establishment realizes it or not, the new generation of coach-proof tests that will be used to evaluate teachers, appear to be measuring the skills students are expected to be learning.  And by teaching what should be taught, teachers are teaching to the test, whether the Gates Foundation and its look-alikes realize it, like it, or not.

BIO: Barry Garelick is an analyst for a federal agency and is cofounder of the U.S. Coalition for World Class Math. (http://usworldclassmath.webs.com/ )  He plans to teach math later this year.

9 Comments | instructional reform | Tagged: , | Permalink
Posted by Jay P. Greene

The Dead End of Scientific Progressivism

January 18, 2011

In Education Myths I argued that we needed to rely on science rather than our direct experience to identify effective policies.  Our eyes can mislead us, while scientific evidence has the systematic rigor to guide us more accurately.

That’s true, but I am now more aware of the opposite failing — believing that we can resolve all policy disputes and identify the “right way” to educate all children solely by relying on science.  Science has its limits.  Science cannot adjudicate among the competing values that might attract us to one educational approach over another.  Science usually tells us about outcomes for the typical or average student and cannot easily tell us about what is most effective for individual students with diverse needs.  Science is slow and uncertain, while policy and practice decisions have to be made right now whether a consensus of scientific evidence exists or not.  We should rely on science when we can but we also need to be humble about what science can and can’t address.

I was thinking about this while reflecting on the Gates Foundation’s Measuring Effective Teachers Project.  The project is an ambitious $45 million enterprise to improve the stability of value-added measures while identifying effective practices that contribute to higher value-added performance.  These are worthy goals.  The project intends to advance those goals by administering two standardized tests to students in 8 different school systems, surveying the students, and videotaping classroom lessons.

The idea is to see if combining information from the tests, survey, and classroom observations could produce more stable measures of teacher contributions to learning than is possible by just using the state test.  And since they are observing classrooms and surveying students, they can also identify certain teacher practices and techniques that might be associated with greater improvement.  The Gates folks are using science to improve the measures of student progress and to identify what makes a more effective teacher.

This is a great use of science, but there are limits to what we can expect.  When identifying practices that are more effective, we have to remember that this is just more effective for the typical student.  Different practices may be more effective for different students.  In principle science could help address this also, but even this study, with 3,000 teachers, is not nearly large enough to produce a fine-grained analysis of what kind of approach is most effective for many different kinds of kids.

My fear is that the researchers, their foundation-backers, and most-importantly, the policymaker and educator consumers of the research are insensitive to these limitations of science.  I fear that the project will identify the “right” way to teach and then it will be used to enforce that right way on everyone, even though it is highly likely that there are different “right” ways for different kids.

We already have a taste of this from the preliminary report that Gates issued last month.  Following its release Vicki Phillips, the head of education at the Gates Foundation, told the New York Times: “Teaching to the test makes your students do worse on the tests.”  Science had produced its answer — teachers should stop teaching to the test, stop drill and kill, and stop test prep (which the Gates officials and reporters used as interchangeable terms).

Unfortunately, Vicki Phillips mis-read her own Foundation’s report.  On p. 34 the correlation between test prep and value-added is positive, not negative.  If the study shows any relationship between test prep and student progress, it is that test prep contributes to higher value-added.  Let’s leave aside the fact that these were simply a series of pairwise correlations and not the sort of multivariate analysis that you would expect if you were really trying to identify effective teaching practices.  Vicki Phillips was just plain wrong in what she said.  Even worse, despite having the error pointed out, neither the Gates Foundation nor the New York Times has considered it worthwhile to post a public  correction.  Science says what I say it says.

And this is the greatest danger of a lack of humility in the application of science to public policy.  Science can be corrupted so that it simply becomes a shield disguising the policy preferences of those in authority.  How many times have you heard a school official justify a particular policy by saying that it is supported by research when in fact no such research exists?  This (mis)use of science is a way for authority figures to tell their critics, “shut up!”

But even if the Gates report had conducted multivariate analyses on effective teaching practices and even if Vicki Phillips could accurately describe the results of those analyses, the Gates project of using science to identify the “best” practices is doomed to failure.  The very nature of education is that different techniques are more effective in different kinds of situations for different kinds of kids.  Science can identify the best approach for the average student but it cannot identify the best approach for each individual student.  And if students are highly varied in their needs, which I believe they are, this is a major limitation.

But as the Gates Foundation pushes national standards with new national tests, they seem inclined to impose the “best” practices that science identified on all students.  The combination of Gates building a national infrastructure for driving educator behavior while launching a gigantic scientific effort to identify the best practices is worrisome.

There is nothing wrong with using science to inform local practice.  But science needs markets to keep it honest.  If competing educators can be informed by science, then they can pick among competing claims about what science tells us.  And they can learn from their experience whether the practices that are recommended for the typical student by science work in the particular circumstances in which they are operating.

But if the science of best educator practice is combined with a national infrastructure of standards and testing, then local actors cannot adjudicate among competing claims about what science says.  What the central authorities decide science says will be infused in the national standards and tests and all must adhere to that vision if they wish to excel along these centralized criteria.  Even if the central authority completely misunderstands what science has to say, we will all have to accept that interpretation.

I don’t mean to be overly alarmist.  Gates has a lot of sensible people working for them and there are many barriers remaining before we fully implement national standards and testing.  My concern is that the Gates Foundation is being informed by an incorrect theory of reform.  Reform does not come from science identifying the right thing to do and then a centralized authority imposing that right thing on everyone.  Progress comes from decentralized decision-makers having the freedom and motivation to choose among competing claims about what is right according to science.

(edited for typos)

12 Comments | instructional reform, methodology | Tagged: , | Permalink
Posted by Jay P. Greene

False Claim on Drill & Kill

December 13, 2010

The Gates Foundation is funding a $45 million project to improve measures of teacher effectiveness.  As part of that project, researchers are collecting information from two standardized tests as well as surveys administered to students and classroom observations captured by video cameras in the classrooms.  It’s a big project.

The initial round of results were reported last week with information from the student survey and standardized tests.  In particular, the report described the relationship between classroom practices, as observed by students, and value-added on the standardized tests.

The New York Times reported on these findings Friday and repeated the following strong claim:

But now some 20 states are overhauling their evaluation systems, and many policymakers involved in those efforts have been asking the Gates Foundation for suggestions on what measures of teacher effectiveness to use, said Vicki L. Phillips, a director of education at the foundation.

One notable early finding, Ms. Phillips said, is that teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains than those who simply work their way methodically through the key concepts of literacy and mathematics. (emphasis added)

I looked through the report for evidence that supported this claim and could not find it.  Instead, the report actually shows a positive correlation between student reports of “test prep” and value added on standardized tests, not a negative correlation as the statement above suggests.  (See for example Appendix 1 on p. 34.)

The statement “We spend a lot of time in this class practicing for [the state test]” has a correlation of  0.195 with the value added math results.  That is about the same relationship as “My teacher asks questions to be sure we are following along when s/he is teaching,” which is 0.198.  And both are positive.

It’s true that the correlation for “Getting ready for [the state test] takes a lot of time in our class” is weaker (0.103) than other items, but it is still positive.  That just means that test prep may contribute less to value added than other practices, but it does not support the claim that  “teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains…”

In fact, on page 24, the report clearly says that the relationship between test prep and value-added on standardized tests is weaker than other observed practices, but does not claim that the relationship is negative:

The five questions with the strongest pair-wise correlation with teacher value-added were: “Students in this class treat the teacher with respect.” (ρ=0.317), “My classmates behave the way my teacher wants them to.”(ρ=0.286), “Our class stays busy and doesn’t waste time.” (ρ=0.284), “In this class, we learn a lot almost every day.”(ρ=0.273), “In this class, we learn to correct our mistakes.” (ρ=0.264) These questions were part of the “control” and “challenge” indices. We also asked students about the amount of test preparation they did in the class. Ironically, reported test preparation was among the weakest predictors of gains on the state tests: “We spend a lot of time in this class practicing for the state test.” (ρ=0.195), “I have learned a lot this year about the state test.” (ρ=0.143), “Getting ready for the state test takes a lot of time in our class.” ( ρ=0.103)

I don’t know whether something got lost in the translation between the researchers and Gates education chief, Vicki Phillips, or between her and Sam Dillon at the New York Times, but the article contains a false claim that needs to be corrected before it is used to push changes in education policy and practice.

UPDATE —

The LA Times coverage of the report contains a similar misinterpretation: “But the study found that teachers whose students said they “taught to the test” were, on average, lower performers on value-added measures than their peers, not higher.”

Try this thought experiment with another observed practice to illustrate my point about how the results are being mis-reported…  The correlation between student observations that “My teacher seems to know if something is bothering me” and value added was .153, which was less than the .195 correlation for “We spend a lot of time in this class practicing for [the state test].”  According to the interpretation in the NYT and LA Times, it would be correct to say “teachers who care about student problems tend to have lower value-added learning gains than those who spend a lot of time on test prep.”

Of course, that’s not true.  Teachers caring about what is bothering students is positively associated with value added just as test prep is.  It is just that teachers caring is a little less strongly related than test prep.  Caring does not have a negative effect just because the correlation is lower than other observed behaviors.

(edited for typos)

3 Comments | instructional reform, journalism, school accountability, teacher quality | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Are National Standards Conservative?

July 23, 2010

Checker Finn and Mike Petrilli seem to think so.  As part of their Gates-fueled pro-standards juggernaut, they have a piece on National Review Online arguing that conservatives should support the current national standards effort.  They write:

Conservatives generally favor setting a “single standard” for everybody. Setting different standards for different people — think affirmative action, for instance — is an idea most associated with the Left.

If by “conservative” we mean people who think that decisions should be decentralized, Finn and Petrilli have it exactly backwards.  National standards are a centrally-imposed, one-size-fits-none approach that would make most conservatives shudder.

Let’s be clear — national standards are being centrally imposed because states are financially punished if they don’t adopt them because they would receive lower scores on their Race to the Top proposals and almost certainly lose out on getting their share of those tax dollars.  National standards are “voluntary” in the same way that federal highway funds are voluntary.  You can disobey the federal dictate as long as you don’t mind having the tax dollars your residents pay go to other states.

Let’s also be clear that conservatives do not generally favor a “single standard” for everyone.  Conservatives do not think everyone should meet a single standard of fashion by being required to wear the same clothes.  Nor should everyone be compelled to meet a single standard of nutrition by being required to eat the same foods.  On what basis would we think conservatives would want every school child to be required to learn the same thing at the same time?  To the contrary, conservatives generally favor allowing consumers (of food, clothing, education, or anything else) to decide how best to serve their own needs by having choice among competing providers with differing products.

It’s true that there are some people who are called conservatives who tend to favor centralization over choice and competition, but those people tend to have more of an authoritarian streak than a liberty-loving streak.  It is one of the weaknesses of our language that the same word — conservative — is used to describe both Benito Mussolini and Milton Friedman.  But no one should be fooled into thinking that policies favored by a “conservative” like Mussolini would also be favored by a “conservative” like Friedman.

The real divide here is between people who think that policies are best when decisions are decentralized and choice and competition are enhanced versus people who think that there is a “right way” that should be imposed centrally and should constrain choice and competition.

Nor are Finn and Petrilli accurate when they assert that national standards are being supported broadly by conservatives except for “a half-dozen libertarians who don’t much care for government to start with.” Is the Wall Street Journal editorial page, which came out against national standards, just a handful of libertarian crazies?  Is the Heritage Foundation, which also opposes national standards, just a handful of libertarian nut-jobs?  Or how about the Pioneer Institute?  And look who’s supporting national standards — fine conservatives like the American Federation of Teachers.

Just because the education bureaucracies in a bunch of red states have signed up for national standards doesn’t mean that the idea has conservative support.  It just means that their budgets are really tight and they want to be in the running for federal Race to the Top dollars as well as gobs of Gates “planning” grant dollars.  The fact that there has not been more active conservative opposition can mostly be explained by the speed with which this is being crammed through in the midst of a severe state budgetary crisis.

But conservatives who favor decentralization, choice, and competition should take heart.  Many of those states will change their minds if they don’t get federal dollars to stay on board.  And the grand national coalition for these standards will probably fall apart as the airy-fairy standards are converted into actual practice in the form of national assessments.  We’ll see how well the Linda Darling-Hammond led national assessment, which I can only imagine involves the testing of drum-circle collaboration, suits conservatives like Finn and Petrilli who so far have supported this enterprise.  And with more time and greater imposition on actual practice, rank and file conservatives will become more mobilized in opposition.

There is a risk that the Obama Administration will link larger amounts of federal dollars, like Title I funds, to full adoption of these standards and a national assessment, in which case conservative opposition may be too little too late.  But if the Obama Administration and the AFT do triumph no one will think it will be a conservative victory.

10 Comments | school accountability | Tagged: , , , | Permalink
Posted by Jay P. Greene

Next Entries »