The Wagner Epic Continues

February 3, 2009

No, not that Wagner.  There is more on Tony Wagner, the snake-oil salesman educational consultant.  My op-ed on Wagner ran in the Northwest Arkansas Morning News.  I’ve also reprinted the text below, since it is easier to read that than the scanned pdf in the link.

The first community discussion on Wagner’s book, The Global Achievement Gap, was held last night.  It wasn’t too bad.  A number of teachers (at tables other than mine) expressed resentment at the suggestion that they weren’t already aware that critical thinking and creativity were desirable.  But administrators and GT teachers seemed more enamored with the book.  And the reaction from parents and community members included a fair degree of skepticism. 

It’s hard to get people to think critically about people who push a focus on critical thinking.  To be for critical thinking is like being for goodness and light.  The tricky part is in how you get there.  To the extent that Wagner has any concrete suggestions, he seems to be taking folks down the wrong path.  He wants less emphasis on content and less testing.  But he shows no evidence that higher levels of critical thinking can be found in places or at times when there was less content and less testing.  In fact, the little evidence he does provide would suggest the opposite.

Some smart folks are pushing back against these data-free educational consultants.  Sandra Stotsky had an op-ed on Wagner last week.  Dan Willingham had an excllent blog post on Alfie Kohn as did Stuart Buck.  And Robert Pondiscio at Core Knowledge ,  AndyRotherham at Eduwonk , and Ken De Rosa at D-Ed Reckoning have added their two cents (which, with the new stimulus package, will become 2 trillion cents).

So here is my op-ed pasted below:

Fayetteville Public Schools Need Evidence, Not Snake-Oil (submitted title)
By Jay P. Greene

              The Fayetteville Public Schools purchased 2,000 copies of Tony Wagner’s The Global Achievement Gap and organized a series of public fora to discuss how that book might guide our schools.  The District is to be commended for engaging the community in this process.  But it is unclear why the District selected Wagner’s book as the focus of this discussion.

Wagner’s book makes claims about what skills students really need to learn, what is blocking them from learning those skills, what countries are more successful in teaching these skills, and what some schools are doing to remedy the problem.  But he provides no systematic evidence to substantiate any one of these claims.  In short, the book is a series of anecdotes that more closely resembles what one would find in a self-help manual than in a work of social science.  If we apply our critical thinking skills, which Wagner says are essential, we should reject this book as a sound basis for planning the future of Fayetteville schools.

First, Wagner says there are seven essential survival skills that our children need to learn.  How does he know that these are the essential skills?  He chatted with a CEO on an airplane and selected a few more to interview.  Does he review any research on the types of skills that predict who will become successful adults?  No. Wagner relies upon the authority of his experience and the experiences of a handful of corporate executives to identify the essential skills.  Accepting claims on this basis would be the sort of thing we would hope people with critical thinking skills might reject.

Frankly, the seven skills he lists — critical thinking, collaboration, adaptability, initiative, communication, analysis, and imagination – seem reasonable enough, but they are also so vague as to be unhelpful in informing schools about what to do.  How exactly do we produce critical thinking or adaptability or creativity?  It’s not as if educators have been unaware of these goals, but they haven’t generally been effective at developing strategies to achieve them.

Then Wagner identifies what he believes is blocking the acquisition of these seven essential skills – high stakes testing.  What evidence does he present to support this claim?  Again, he presents no systematic evidence to demonstrate that there is a tradeoff between the content knowledge required in accountability testing and the essential skills he wants.  Couldn’t it be the case that improving mastery of basic skills and content knowledge provides the foundation for these seven skills?  It’s hard to be imaginative, analytical, etc… without knowing subject matter and basic skills of literacy and numeracy.  Einstein may have said “imagination is more important than knowledge,” as the book’s dedication indicates, but Einstein couldn’t have succeeded without a firm grasp of advanced mathematics.

If Wagner were right that accountability testing undermines essential skills, then surely these skills must have been more plentiful before testing became as salient as it is today.  But Wagner does not (and cannot) provide any evidence to show that.  Instead, he shows (on p. 74) that students in the United States significantly lag students in Finland, Hong Kong-China, Japan, and Korea in certain problem solving skills on an international test called PISA.  As it turns out, high stakes testing is extremely prominent in most of these countries with strong problem-solving results – a fact curiously at odds with Wagner’s claims.  If accountability testing undermines essential skills, why do countries with such strong accountability systems manage to succeed so well in teaching the essential skills Wagner wants?

Wagner describes three model schools that he says have been effective at teaching essential skills (although he again fails to provide any evidence that they are as successful as he claims).  But it is by no means clear that the approaches adopted by these three schools are the only valid approaches or that they could be replicated easily by others.  Replication is especially problematic because the three models he provides are all charter schools or alternative schools of choice.  Perhaps the secret of these schools’ success has something to do with school choice and not the features he describes.  If true, it’s not clear how Fayetteville could imitate the success of these schools.

To achieve our goals in education we have to adopt approaches backed by systematic evidence.  If we believe critical thinking, collaboration, and creativity are the most important goals for schools, then we need systematic evidence on systems of teacher preparation, curriculum, and pedagogy that effectively produce those goals.  There is a growing body of scientific research on these issues, including a number of studies sponsored by the U.S. Department of Education’s Institute for Education Sciences, that the Fayetteville Public Schools might wish to consider rather than consulting with the latest peddler of educational snake-oil.

10 Comments | instructional reform, school accountability | Tagged: , , | Permalink
Posted by Jay P. Greene

Building Without Foundation of Evidence

February 1, 2009

This is the model for new school construction. (HT: Arkansas Project)

 I understand why House Democrats included $20 billion for school construction in the $819 billion stimulus package they passed last week.  They need to throw money out the window as fast as the printing presses can make it.  That way they can say they are doing something to “help” the economy.  And as long as they are doing something they might as well help their friends in the educational industrial complex.

What I don’t understand is why some normally smart people feel compelled to justify this school construction spending by claiming that it will help students or that it is badly needed.  There is no evidence to support these claims.  That’s why I was surprised to see Sara Mead, senior research fellow at the New America Foundation, write:

“But perhaps the most important reason to invest in school construction is that our students need it. Just as Americans have underinvested in our bridges, roads, and other infrastructure, we’ve also underinvested in our education infrastructure. The American Society of Civil Engineers Infrastructure report card gives our school buildings a grade of D — lower than grades for bridges, rail, or public transit infrastructure. According to the National Center for Education Statistics, it would cost $127 billion just to renovate and repair crumbling or outdated school facilities to good condition. Poor school facilities don’t necessarily prevent students from learning, but, it’s unconscionable that we currently ask students to learn, and teachers to work, in buildings that are overcrowded, inadequately heated and ventilated, poorly maintained, and in some cases literally falling apart. The contrast between schools and other buildings sends our most disadvantaged children a devastating message about the value we place on their education.”

As readers of JPGB may recall, the research literature finds no relationship between school facilities and student achievement (outside of developing countries, where a grass hut and mud floor may be a hindrance in bad weather).  To repeat: “In the Handbook of the Economics of Education, Eric Hanushek reviews all of the research meeting minimal quality standards regarding the relationship between school facilities and student performance.  He identifies 91 analyses on the issue in the U.S. and finds that 86% of them show no statistically significant relationship.  Of the remaining 14% of analyses that did show significant effects, 9% were positive and 5% were negative. ”

If the evidence generally fails to find that better school facilities improve student outcomes, it’s not clear how Sara Mead can support claims like “our students need it” or that current facilities send “a devastating message about the value we place on their education.”  If it has no effect on their learning, how devastating could it be?

Perhaps Sara Mead relies upon the assessments from the American Society of Civil Engineers (ASCE) and the National Center for Education Statistics (NCES) to support her claims.  Let’s look into those references a little further to see whether we should believe them or believe the research reviewed by Rick Hanushek.

The ASCE grade of D-  for school infrastructure is in a report card issued by the interest group representing engineers who design public infrastructure.  To determine the grades ASCE “assembled a panel of 24 of the nation’s leading civil engineers” who then reviewed other reports and surveys of civil engineers.  I would be more specific about their methodology, but they provide almost no information about their methods or standards.  Their entire description of the methodology consists of 5 sentences.  You can read it yourself to see.

In the 2009 report card they actually gave school infrastructure a D.  But don’t worry, that is the average grade they gave to the 15 categories of public infrastructure they graded, which ranged from C+ to D -.  They also determined that we need to spend $2.2 trillion (with a T!) on public infrastructure.  How exactly did they determine that?  Again, the five sentences describing their methodology didn’t exactly explain all that.  But since they are a lobby group representing engineers who work on public infrastructure projects, I’m sure we can just trust their expert judgment that we urgently need to spend $2.2 trillion on what they do.

Using the NCES to support a need for $127 billion in school construction spending is hardly more persuasive.  To be precise, the NCES did not determine the amount that needs to be spent on schools.  They just surveyed school district officials and in the survey asked the school officials how much money they thought needed to be spent on their schools to bring them to “good” condition.  The NCES does not represent this result as their own opinion; they portray it as the opinion of the people they surveyed.

Again, asking people who are the potential beneficiaries of public spending whether they need more money without any standards to define their “need” is unlikely to be a reliable method. 

To illustrate the point, I’ve conducted a survey of “experts” on JPGB – the regular contributors — to determine the resources we would need from the federal bailout to bring the blog “into good overall condition.”  One respondent identified the need for martinis, cigars, and a few Vegas junkets to improve our creative process.  Another identified the need for a board game marathon only interrupted by Indian food buffet binges to sharpen our intellect.  And another agreed with the first two and added  the need for a spaceship, pony, and a lifetime supply of spicy wings and candy.  (You can guess who’s who.)

I think I’m going to trust Hanushek’s assessment of 91 analyses that meet social science standards over the self-serving assessments of school officials and and a construction lobbying organization.  If you disagree, then I expect that you’ll support our demands to add JPGB to the bailout since, after all, the “evidence” clearly demonstrates our need.

18 Comments | money myth | Tagged: | Permalink
Posted by Jay P. Greene

Blaming Special Ed

January 4, 2009

It’s all too common but also completely mistaken to blame special education for the shortcomings of the public k-12 system.  If you point out that per pupil spending has more than doubled in the last three decades (adjusting for inflation) while student outcomes have remained unchanged, people blame the rising costs of special education.  (See for example Richard Rothstein on this).  If you point out that the teaching workforce has increased by about 40% in the last three decades (adjusted for changes in student population), people blame special education (see below).  If budgets are tight and programs get cut, people blame special education for draining money from general education. 

Blaming special ed is easy.  Most attempts to blame special ed don’t even bother presenting data or make the most crude use of data to support their claims.  Reporters simply accept assertions from school and state officials without question.  Folks accept the blame-special-ed-story so easily because — well, to put it bluntly — it is a a widely held but unstated prejudice.  People quietly resent special education because they fear that it is short-changing their regular education students.  They assume that money spent on disabled kids is necessarily money taken away from general education.  They can’t imagine that resources for general education have also increased at a very rapid clip even as special ed costs have risen. 

School officials — people who should know better — play upon this popular prejudice to rationalize their failures.  They would never dare blame the programs that have been created or expanded in the last three decades for the education of poor and minority students.  Those programs also cost quite a lot of money.  No, school officials choose to blame special ed because it seems like blaming fate.  Fate has overwhelmed us with a rise in disabilities, the story goes, which have drained general education of money, teachers, and flexibility under tight budgets.  Never mind the considerable evidence that the rise in special education over the last few decades is almost certainly due to an increased classification of students as disabled rather than a true increase in the rate of disabilities in the world.  Fate had nothing to do with it.

I’ve rebutted the claims that special ed is largely responsible for rising per pupil spending in chapters 1 and 2 of the book Education Myths as well as in this Education Next article and in this paper that was published in the Peabody Journal of Education. 

My purpose in this post is to address the comment written by “Kevin” that attributed the increase in the teaching workforce to special education.  Kevin was responding to a post by Greg in which he wrote: “But teachers’ unions have pushed up costs – dramatically. In the past 40 years, the cost of the government school system per student has much more than doubled (even after inflation) while outcomes are flat across the board. And this has mainly been caused by a dramatic increase in the number of teachers hired per student – a policy that benefits only the unions.”  And Kevin replied: “Any comparison of staffing in schools 40 years ago and today typically ignores one group of staff that didn’t exist in 1968 – special education teachers and aides. Special education programs weren’t in most schools 40 years ago, hence there were no staff hired to work with those specific populations, particularly students with cognitive delay and autism who need a much higher staff ratio than is provided in the general education classroom.”

I’m bothering to rebut Kevin’s claims because 1) he appears to be a state employee (perhaps a school official, judging from his email address), and 2) his comments are typical of the blame special ed rhetoric.  Notice that Kevin doesn’t bother to present any evidence.  He just tells a plausible story, which because he and many others have “pre-judged” it to be true, they consider persuasive without need of any proof.  But let’s consider the evidence here.

In 1974, the year before federal legislation governing the education of disabled students was adopted, there were 2.165 million public school teachers and an average student to teacher ratio of 20.8.  In 2006 there were 3.177 million public school teachers, an increase of 1.012 million teachers.  And in 2006 there were a total of 404,577 teacher FTEs providing special education services.

But we have to adjust for the fact that that some of those 404,577 teachers assigned to special education have been shifted (or had their lines shifted) to special education as more students have been reclassified as disabled.  We also have to adjust for the fact that there are more students in 2006 than in 1974.  To make everything comparable, let’s assume that the student-teacher ratio had remained at 20.8 for all students.  Given that there were 49.370 million public school students in 2006, there would have been 2.374 million teachers if ratios had stayed the same for everybody instead of the 3.177 million teachers we have.  So there was really an increase of 803,442 teachers, adjusted for the change in student population.

But if the 6,081,890 students classified as disabled also had 20.8 students for each teacher, they would have 292,398 teachers.  Given that there are 404,575 teachers assigned to special education, the lower student-teacher ratios required for disabled students only results in a net increase of 112,179 teachers (404,575 minus 292,398).  So, of the 803,442 teachers added since 1974 only about 112,179 can be explained by the need to offer smaller student-teacher ratios to disabled students.  That is, special ed may only account for about 14% of the increase in the teaching workforce.

What people like Kevin forget is that while virtually “no staff” were hired specifically for special education several decades ago, there were also virtually no students classified as disabled (although most were in schools and under-served).  If we shift 6 million students into special education and maintained the same 1974 ratio of 20.8 students per teacher, we would have shifted 292,398 teachers with them.  It’s true that with an increase in federal and state subsidies along with a mandate to provide services, we’ve reduced student teacher ratios for disabled students.  But we’ve only added an additional 112,179 teachers to produce smaller ratios for disabled students.

Of course, Greg also makes an excellent point when he says in the comments to his post that resources devoted to special ed should also be expected to produce improvements in results.  Regardless of how resources have been allocated between regular and special education, the money should be yielding benefits for students.  The fact that we have observed virtually no change in student outcomes over the last four decades despite a huge increase in real expenditures (regardless of how it was allocated) is a source of chronic frustration with public education. 

The unstated assumption of these blame-special-ed stories is that money spent on special education is basically money flushed down the toilet.  They assume that nothing can help disabled kids, which fuels the quiet resentment of resources devoted to special education.  Rather than looking for scapegoats — special education, rising poverty, cosmic rays, etc… — folks should focus on the perverse incentives of a broken public education system.  The fault, dear reader, lies not in our stars but in ourselves.

22 Comments | myth of helplessness, special ed | Tagged: , , , , | Permalink
Posted by Jay P. Greene

Only Mostly Dead

November 17, 2008

princessbride11

Reliance on markets and the idea of limited government are not quite dead — only mostly dead.  They (mostly) died on October 3, 2008 when Congress passed the ginormous (giant + enormous) bailout bill, greatly expanding the scope and authority of the federal government to own stakes in businesses and financial assets.  And if you are looking for accomplices in the (mostly) murder of market-reliance and limited government, you should probably investigate the DC based “market-oriented” think tanks.

George Will correctly warns that this expansion of government by the partial nationalization of large sections of our economy is unlikley to be either temporary or benign.  (Now he tells us!)

The way back from (mostly) death for supporters of markets and limited government is to undo the bailout as quickly as possible.  Let businesses that made unwise decisions go into bankruptcy (I’m looking at you, GM).  Let their assets be reorganized by their credit-holders so that they move forward with a more efficient structure and more competent management.  Unless people experience the consequences of their mistakes, they can never learn from those mistakes and do better in the future.

I made this exact argument in defense of allowing companies to fail on September 18.  Mike Petrilli made the same argument on the same day.  But we’re just a bunch of lowly education analysts.  Where were all of the limited government Republicans?  Where were the market-oriented think tanks?

Let’s take a look at the period between September 15 and September 22 to see what the national, market-oriented think tanks had to say.  Remember that this was the pivotal week that began the (mostly) death-rattles for limited government and market-reliance.  Lehman Brothers was allowed to go bankrupt on September 15, but AIG received its first $85 billion bailout offer on September 16.  The first proposed $700 billion bailout was circulated around midnight on September 20. 

This was the time when the folks at think tanks could have been standing athwart big government yelling STOP!  They could have bolstered anti-bailout Republicans in Congress, steered the McCain campaign against the bailout (which would have been risky but probably a better hail Mary pass than picking Palin as VP), and they could have laid the foundations for a future defense of markets and limited government.

For the most part, the “market-oriented” national think tanks failed to yell STOP.  In the culminating act of complicity with big-government conservatism, they rationalized and defended a large government intervention in the economy.

Here is what people affiliated with AEI wrote during that period:

Glenn Hubbard called for a Resolution Trust approach of a bailout on both September 15 and September 19, advocating “putting in place a clean-up agency like the 1930s’ Homeowner’sLoan Corp. or the 1980s’ Resolution Trust Corp. would help…. The fiscal costs of inaction would be significant, both in lost tax receipts and in larger ‘crisis’ bailouts down the road.”  This Resolution Trust idea was the foundation for the $700 billion bailout plan of September 20.

Lawrence Lindsey called for a lifting of any cap on depositor insurance at banks on September 17 and then on September 21 endorsed the idea that the government had to provide credit to distressed financial institutions: “But by far the most inevitable economic development will be an expansion of the balance sheets of the government and its central bank.  When credit bubbles burst an enormous hole is formed in private-sector balance sheets…. Government, and only government, inevitably fits the bill as it can both tax and print the resources it needs.”  More support for the bailout.

Vincent Reinhart urged the administration to have “backbone” and resist more bailouts on September 16, but by September 22 he wrote in the New York Times: “The Congress should authorize the Treasury to purchase asset-backed securities in the secondary market and mortgages through auctions. For assets where it might not have all the information it needs, the Treasury could demand a slice of equity in the selling firm as well.”  More support for the bailout.

Alex Pollock wrote on September 17: “When government financial officers–like Treasury secretaries, finance ministers and central-bank chairmen–stand at the edge of the cliff of market panic and stare down into the abyss of potential financial chaos, they always decide upon government intervention.  This is true of all governments in all countries in all times. Nobody is willing to take the chance of going down in history as the one who stood there and did nothing in the face of a financial collapse and debt deflation. Put in their place, you would make the same decision, and so would I.”  More support for bailout.

Desmond Lachman wrote on September 17: “If Main Street is to be spared the painful economic consequences of a financial market meltdown on Wall Street, policymakers have little alternative but to resort to unorthodox interventionist policies to put a floor under the housing market and to prop up the banks with taxpayers’ money.”  More support for the bailout.

Newt Gingrich, writing on September 21, took a very skeptical position on the bailout.  But David Frum strongly went in the other direction, writing on September 22: “What should a free-market believer think about the plan for a government bailout of the U.S. mortgage market? Try this analogy: You have a white carpet in your upstairs hall. The normal rule is that nobody can wear shoes on the carpet. But the house is on fire–and the baby is upstairs. Will you tell the arriving fire department to wait and kick their boots off before dousing the flames?”  Notice that in this analogy, reliance on markets and belief in limited government are just the aesthetic nicety of clean carpets, not the principles that lay the foundations for the house or materials that resist fire.

It’s true that some of these folks called for “smarter regulation” (don’t make me get all Dr. Evil on them!) and advised about how best to conduct a bailout. But the bottom line is that 6 out of 7 AEI fellows who wrote during the pivotal week of September15-22 came out in support of a government bailout, with the 7th expressing skepticism but not outright opposition.

What about The Heritage Foundation?  They supported the bailout.

They issued four policy briefs during the week Sept 15-22.  The pieces all had the same basic message in support of a bailout: “Congress needs to act carefully but quickly in passing this legislation, knowing that it can correct any flaws when it reconvenes next year. Quick action is needed because financial markets remain deeply stressed, and the stress continues to spread to the rest of the economy.”

And what about the Manhattan Institute? They didn’t support the bailout.

Nicole Gelinas expressed even more doubts about the bailout than Newt Gingrich.  Writing in the NY Post she said, “Thing is, it’s not clear this is a solution. There’s no guarantee that even this much cash can buy us out of a systemic financial crisis. Even if it does, we probably face years of necessary financial and economic readjustment.”  And on September 26, just outside of the time period we are examining, she began to actively oppose the bailout, worrying that it might actually delay recovery.

And how about Cato? They also didn’t support the bailout.

Cato behaved more in-line with expectations than AEI or Heritage.  A September 15 piece by James Dorn was typical:“When the US Treasury is raided to defend the government’s credibility to guarantee GSE debt, it may calm markets for a time. Yet, in the long run, the drifts towards socialism and increased government borrowing requirements discourage foreign investment, decrease private saving, increase interest rates and slow US growth. That is a high price to pay for ‘stability.'”

For those of you keeping score, AEI and Heritage were actively in support of a large government intervention in the economy.  The Manhattan Institute and Cato were not.  But AEI is by far the most active and influential market-oriented think tank on this matter, so their support was crucial in shaping events and contributing to the (mostly) death of limited government and market ideas. The Manhattan Institute had only one expert on economic affairs active during the period I examined.  AEI had seven.  I believe that Heritage has, by far, the largest operating budget of any of these think tanks ($39 million as of 2006).  That is more than three times as large as the Manhattan Institute’s $12 million annual budget.

Donors pay for those seven AEI fellows and provide Heritage with its ginmormous budget.  If those donors really do wish to support huge expansions in government involvement in the economy, then I guess they are giving to the right organizations.

12 Comments | economics | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

The Federal Role In Education

October 21, 2008

Mike Petrilli has an excellent piece on Flypaper about lessons for the next administration on the limits of federal involvement in education policy.  He’s reacting to a report by Sara Mead and Andy Rotherham laying out an agenda for the federal government, which presumably they will be helping administer for an Obama administration.

Having learned these lessons the hard way, Mike warns that Sara and Andy are falling into old traps despite the best of intentions.  Mike argues that giving money to favored organizations, such as KIPP charters and Teach For America to “Grow What Works” will suffer from the same flaws as the Bush administration’s efforts to give money to favored organizations, such as Reading First.  Even if the favored groups are doing great work, giving money to them will be portrayed by opponents and the media as cronyism and pork. 

In addition, Mike notes that expanding Teach For America and KIPP requires cooperation from state and local agencies to lift caps on charters, equalize funding for charters and traditional public schools, and relax certification requirements.  The problem is that state and local agencies have perfected the art of subverting federal mandates.  At best unwilling state and local agencies will minimally comply with federal requirements while eviscerating their spirit.  At worst they will defy the requirements and dare the federal government to withhold funds.  The feds generally lack the political nerve to risk the political fallout from actually applying a sanction to a local or state education agency.

Let me expand Mike’s observations to draw lessons for the future of No Child Left Behind.  Like Mike, I once believed that the federal government could use the carrots and sticks (mostly sticks) in NCLB to motivate local and state education agencies to improve.  Since I was convinced by evidence that incentive systems worked, why shouldn’t the federal government do what works? 

My mistake and the mistake of NCLB was in not considering how much implementation of those incentive systems matters.  The federal education bureaucracy lacks the familiarity with local circumstances, the nimbleness to respond to changing circumstances, and the political will to apply sanctions to properly implement an incentive system.  Incentive systems are good for education reform but the federal government is too big, slow, far-away, stupid, and cowardly to do it right. 

The same is likely to be the case when the federal government tries to expand Teacher For America and KIPP under an Obama administration.  As Andy and Sara will soon discover and as Mike has warned them, the federal government will be obstructed by unwilling local and state actors.  And the mandates the Feds issue to overcome that resistance will trample upon or fail to anticipate local circumstances.

So what can the federal government do right?  First, they can continue to improve the availability of information about the school system.  NCLB deepened and entrenched the testing requirements that 37 states had already adopted before NCLB was adopted.  Improving transparency facilitates better policy evaluation and the development of effective state and local accountability systems.

Second, the federal government can facilitate “redistributive” efforts that localities cannot pursue without being punished by collective action issues.  For example, no locality can operate a substantial special education or English language learner program without attracting more students needing services, which then drives up the costs of the programs and drives away the local tax base that pays for those programs.  (See Paul Peterson’s The Price of Federalism for a great discussion of this).  To the extent that we want redistribution, we need the federal government to mandate it.  And I fully confess that I depart from my Cato colleagues in that I think we need some (but very limited) redistribution.

Third, the federal government can fund pilot programs to experiment with new ideas and approaches.  But I should emphasize that I think the federal government has no business evaluating or paying for evaluations of those efforts.  The evaluation process in the US Department of Education and the small number of contract-research firms is far too politicized to be reliable.  Instead, the federal government should play its role of improving transparency by making data on the pilot programs it sponsors available to any qualified researcher rather than to a favored research firm.  The Feds should heavily be in the data collection and distribution business, much as the Department of Commerce makes economic data available, but they should leave analyses of those data to the market of ideas.

The failures of the Bush administration have been a humbling experience.  But we are doomed to repeat their mistakes if we do not learn from them and limit the federal role in education to what the Feds can actually do well.

(edited for typos)

9 Comments | NCLB | Tagged: , , , , , , | Permalink
Posted by Jay P. Greene

NCLB: Less Than Meets the Eye, More Than Nothing

July 29, 2008

Given all of the wailing and gnashing of teeth, one would think that NCLB was a crushing burden on the school system.

In actuality NCLB demands very little. It only requires that states wishing to receive Title I funds have to establish goals for student success, select tests for measuring progress towards those goals, and report results from those tests broken out by subgroups.

The sanctions for failing to make progress toward those goals are almost non-existent. Schools failing to make progress have to offer tutoring or allow students to transfer to better-performing public schools in the same district (if one can be found). But, as we have previously discussed on this blog and in this article, there is widespread non-compliance with even these minimal sanctions. Too often schools fail to inform parents properly of their options under NCLB or direct students into their own tutoring programs, resulting in very few students taking any resources out of their local school, let alone district. Without placing school funds in jeopardy, the only possible sanction is public embarrassment. And that plus $4 will get you a latte at Starbucks.

I do not believe that a single tenured teacher out of the more than 3 million teachers currently working in public schools has been fired, experienced a pay-cut, or otherwise been meaningfully sanctioned because of NCLB. I do not believe that a single student out of the 50 million enrolled in public schools has been held back a grade, been denied a diploma, or otherwise been meaningfully sanctioned as a result of NCLB. (Some states have retention and graduation requirements as part of their state accountability systems, but those policies are not required for NCLB.) Yes, chronically failing schools might eventually face “restructuring” but that is likely to be yet more bark and no bite. Next they’ll be put on double secret probation.

So what supports complaints about “pressure cooker NCLB testing,” or “NCLB-post traumatic stress disorder,” or other “NCLB outrages”? If NCLB has almost no real consequences for teachers or students, what is all of the fuss about? The overwrought reaction seems to have more to do with a political campaign over the future direction of education policy than the actual effects of the current policy.

The most important future policy that the higher volume of squealing is meant to influence is increasing education spending. A center-piece of the complaints about NCLB is that it is an unfunded mandate. Let’s leave aside the fact that federal spending on education has increased 41% since passage of NCLB. And let’s leave aside that NCLB is not actually a mandate, since states do not have to comply with NCLB if they do not want Title I funds (which have increased 59% since 2001).

Besides neither being unfunded nor a mandate, the argument that NCLB is an unfunded mandate is especially odd because it makes one wonder what all of the funding that schools received before NCLB was for. It’s as if the unfunded mandate crowd is saying: “The $10,000 per pupil we already get just pays for warehousing. If you actually want us to educate kids, that’ll cost ya extra.” Remember, that NCLB just asks states to establish and meet their own goals. Didn’t they have goals before NCLB?

While NCLB demands much less than the overwrought rhetoric about it suggests, it does not demand nothing. Most importantly, NCLB entrenched the idea that we should take regular measures of student achievement and report the results, including results for subgroups. Even this is a smaller thing that it may seem at first glance since 37 states had already adopted state testing and accountability systems before passage of NCLB. But NCLB brought the laggard states on-board to this growing national consensus that we ought to have some systematic measures of how our students are doing. It also made reversal of this growing testing and accountability culture more difficult by placing it in federal as well as state law.

Greg Forster has already made the case for why this shift under NCLB has been important, so I will not repeat it here. I would just emphasize that the controversy over NCLB is not really about what NCLB does, but about the broader policy shift that it represents and the extra funding that folks hope they may get as they acquiesce to that policy shift.

26 Comments | school accountability | Tagged: , , , , | Permalink
Posted by Jay P. Greene

When “Sorry” Means “J’Accuse”

July 28, 2008

The letter "J'accuse"

The following column about a letter appeared on the front page of the Cleveland Plain Dealer on Wednesday, July 23.  The letter is in the form of an apology, but it is actually a series of accusations about testing and accountability.  Like another front page letter of accusation, this one has all of Emile Zola’s moral outrage but has none of Zola’s justification.

I’ve reprinted it here with my comments in blue italics.

Students pass state test, but at what cost to their education?

by Regina Brett

The school report cards came out in June.

Rocky River Middle School passed the 2008 Ohio Achievement Tests, earned an Excellent rating from the state and met the requirements for Annual Yearly Progress.

For all of those accomplishments, Principal David Root has only one thing to say to the students, staff and citizens of Rocky River:

He’s sorry.

Root wants to issue an apology. He sent it to me typed out in two pages, single spaced.

He’s sorry that he spent thousands of tax dollars on test materials, practice tests, postage and costs for test administration.

Actually, he did not spend the money.  The taxpayers did when they decided through their elected representatives to adopt a testing and accountability system.  They then hired David Root to implement this policy in his capacity as principal at a public school.

Sorry that his teachers spent less time teaching American history because most of the social studies test questions are about foreign countries.

I guess the people of Ohio thought it was important for students to learn about foreign countries when they, through their elected representatives and hired agents, devised the state curriculum and test.  Besides, if students learned more about foreign countries they might know who Emile Zola was.

Sorry that he didn’t suspend a student for assaulting another because that student would have missed valuable test days.

Sounds pretty irresponsible.  Would he have made a different decision if the student would have missed valuable instructional days?  If so, whose fault is that?  Oh yes, I forgot that this is an accusation, not an apology.

Sorry he didn’t strictly enforce attendance because all absences count against the school on the State Report Card.

So, is David Root saying that he cheated on the state accountability system?  Isn’t this like lying to your boss about your job performance?  Will he be fired, sanctioned, or resign to make amends for his infraction?

He’s sorry for pulling children away from art, music and gym, classes they love, so they could take test-taking strategies.

Why didn’t he just follow the state curriculum and let the scores show what students knew? The decision to take time away for “test-taking strategies”  was completely unnecessary given that more than 90% of Rocky River students have been scoring above the proficient level in reading, math, and writing.  It sounds like they would have done just fine on the state test without working on test-taking strategies and having spent more time on art, music, and gym.

Sorry that he has to give a test where he can’t clarify any questions, make any comments to help in understanding or share the results so students can actually learn from their mistakes.

How reliable would the results be if principals could clarify questions, help in understanding, or share secure test items that would be re-used on future tests?  Does every assessment have to be a formative assessment?

Sorry that he kept students in school who became sick during the test because if they couldn’t finish the test due to illness, the student automatically fails it.

This sounds like a difficult decision.  Football coaches similarly have to think about whether to take injured players out of the game versus having the players tough it out.  We pay leaders to make these difficult decisions, balancing competing interests wisely.

Sorry that the integrity of his teachers is publicly tied to one test.

Actually, the state accountability system — let alone “the integrity of his teachers” —  is not based on one test.  The overall rating of Rocky River Middle School is based on several test results (in Reading, Math, Writing, Social Studies, and Science), the progress students have made in those subjects, and (as we already heard about) the possibly fraudulent attendance rate.

He apologized for losing eight days of instruction due to testing activities.

I thought Root didn’t want one test, so it takes time to administer several.  While testing takes place on eight days it does not (or at least does not have to) consume the entirety of those days.  My understanding is that the average student only spent two mornings being tested, as testing occurred in different grades and subjects for different students across eight days.

For making decisions on assemblies, field trips and musical performances based on how that time away from reading, math, social studies and writing will impact state test results.

I would hope that the principal would think about how assemblies, field trips, and musical performances impact instructional time for other academic subjects regardless of whether those subjects are part of a state accountability system.

For arranging for some students to be labeled “at risk” in front of their peers and put in small groups so the school would have a better chance of passing tests.

Again, if smaller group instruction would help certain students, the principal should arrange for that regardless of the state accountability system.  And the principal would have to think of a way to provide that necessary assistance without stigmatizing the students who need it.

For making his focus as a principal no longer helping his staff teach students but helping them teach test indicators.

Why didn’t he just help his staff teach the subjects with confidence that the test indicators would show what they had learned?  This is especially puzzling given how likely it is that students at Rocky River would pass the state test without paying any special attention to test-taking strategies.

Root isn’t anti-tests. He’s all for tests that measure progress and help set teaching goals. But in his eyes, state achievement tests are designed for the media to show how schools rank against each other.

Seems like the state accountability system does measure progress and help set teaching goals.  What’s wrong with it also informing the public and policymakers (via the media) about how their schools are doing?

He’s been a principal for 24 years, half of them at Rocky River Middle School, the rest in Hudson, Alliance and Zanesville. He loves working with 6th, 7th and 8th graders.

“I have a strong compassion for the puberty stricken,” he joked.

His students, who are 11, 12, 13 and 14, worry that teachers they love will be let go based on how well they perform.

One asked him, “If I don’t do well, will you fire my teacher?”

He cringed when he heard one say, “I really want to do well, but I’m not that smart.”

Has a single tenured teacher in Ohio (or in the United States) been let go based on performance on state accountability tests?  Maybe he should reassure the students that their concern is misplaced.

He wants students to learn how to think, not take tests.

Can’t they do both?

“We don’t teach kids anymore,” he said. “We teach test-taking skills. We all teach to the test. I long for the days when we used to teach kids.”

Why not just return to those days and let the test results show what kids have learned?

Unless we get back to those days, principals and teachers all over Ohio will continue to spend your tax dollars to help students become the best test takers they can be.

The people of Ohio decided to adopt an accountability system because the schools weren’t doing an adequate job teaching kids to think without it.  The “just trust us to do a good job” approach wasn’t working.

(edited to add color)

7 Comments | school accountability | Tagged: , , , , , , , | Permalink
Posted by Jay P. Greene

Eduwonkette Apologizes

July 8, 2008

I appreciate Eduwonkette’s apology posted on her blog and in a personal email to me.  It is a danger inherent in the rapid-fire nature of blogging that people will write things more strongly and more sweeping than they might upon further reflection.  I’ve already done this on a number of occasions in only a few months of blogging, so I am completely sympathetic and un-offended.

One could argue that these errors demonstrate why people shouldn’t write or read blogs.  In fact some people have argued that ideas need a process of review and editing before they should be shown to the public.  These people tend to be ink-stained employees of “dead-tree” industries or academia, but they have a point: there are costs to making information available to people faster and more easily.

Despite these costs the ranks of bloggers and web-readers have swelled.  There are even greater benefits to making more information available to more people, much faster than the costs of doing so.  People who read blogs and other material on the internet are generally aware of the greater potential for error, so they usually have a lower level of confidence in information obtained from these sources than from other sources with more elaborate review and editing processes.  Some material from blogs eventually finds its way into print and more traditional outlets, and readers increase their confidence level as that information receives further review.

Of course, the same exact dynamics are at work in the research arena.  Releasing research directly to the public and through the mass media and internet improves the speed and breadth of information available, but it also comes with greater potential for errors.  Consumers of this information are generally aware of these trade-offs and assign higher levels of confidence to research as it receives more review, but they appreciate being able to receive more of it sooner with less review.

In short, I see no problem with research initially becoming public with little or no review.  It would be especially odd for a blogger to see a problem with this speed/error trade-off without also objecting to the speed/error trade-offs that bloggers have made in displacing newspapers and magazines.  If bloggers really think ideas need review and editing processes before they are shown to the public, they should retire their laptops and cede the field to traditional print outlets. 

We have a caveat emptor market of ideas that generally works pretty well.

So it was disappointing that following Eduwonkette’s graceful apology, she attempted to draw new lines to justify her earlier negative judgment about our study released directly to the public.  She no longer believes that the problem is in public dissemination of non-peer-reviewed research.  She’s drawn a new line that non-peer-reviewed research is OK for public consumption if it contains all technical information, isn’t promoted by a “PR machine,” isn’t “trying to persuade anybody in particular of anything,” and is released by trustworthy institutions.

The last two criteria are especially bothersome because they involve an analysis of motives rather than an analysis of evidence.  I defended Eduwonkette’s anonymity on the grounds that it doesn’t matter who she is, only whether what she writes is true.  But if Eduwonkette believes that the credibility of the source is an important part of assessing the truth of a claim, then how can she continue to insist on her anonymity and still expect her readers to believe her.  How do we know that she isn’t trying to persuade us of something and isn’t affiliated with an untrustworthy institution if we don’t know who she is?  Eduwonkette can’t have it both ways.  Either she reveals who she is or she remains consistent with the view that the source is not an important factor in assessing the truth of a claim.

No sooner does Eduwonkette establish her new criteria for the appropriate public dissemination of research than we discover that she has not stuck to those criteria herself.  Kevin DeRosa asks her in the comments why she felt comfortable touting a non-peer-reviewed Fordham report on accountability testing. That report was released directly to the public without full technical information, was promoted by a PR machine, comes from an organization that is arguably trying to persuade people of something and whose trustworthiness at least some people question.

So, she articulates a new standard: releasing research directly to the public is OK if it is descriptive and straightforward.  I haven’t combed through her blog’s archives, but I am willing to bet that she cites more than a dozen studies that fail to meet any of these standards.  Her reasoning seems ad hoc to justify criticism of the release of a study whose findings she dislikes.

Diane Ravitch also chimes in with a comment on Eduwonkette’s post: “The study in this case was embargoed until the day it was released, like any news story. What typically happens is that the authors write a press release that contains findings, and journalists write about the press release. Not many journalists have the technical skill to probe behind the press release and to seek access to technical data. When research findings are released like news stories, it is impossible to find experts to react or offer ‘he other side,’ because other experts will not have seen the study and not have had an opportunity to review the data.”

Diane Ravitch is a board member of the Fordham Foundation, which releases numerous studies on an embargoed basis to reporters “like any news story.”  Is it her position that this Fordham practice is mistaken and needs to stop?

3 Comments | methodology, research reports | Tagged: , , , , , | Permalink
Posted by Jay P. Greene

Walmart Shareholder Meeting

June 6, 2008

This morning I went to the Walmart shareholder meeting held in the University of Arkansas basketball stadium.  The theme was the new marketing slogan, “Save money.  Live better.”  They presented impressive evidence and compelling anecdotes of how Walmart saves money for families of modest means and, in doing so, improves people’s lives. 

The best example they provided was their $4 prescription drug program.  Lee Scott, the CEO, emphasized that policymakers have been trying to get more people to switch to cheaper generics for years.  But Walmart has been able to succeed where the government has failed, by bringing the price down.  How did they do that when the government hasn’t?  Walmart was able to squeeze the pharmaceutical companies in a way that the government won’t.  Just think of the Medicare Drug Benefit program that has been a near-total flop, failing to make drugs more available while costing taxpayers dearly. 

It struck me that if Walmart were a government program, designed to provide basic goods to low-income families at reduced prices, it would be lauded as a great success on the order of the New Deal or the Marshall Plan.  Books would be written about how it worked so well.  Conferences would be organized to sing its praises.  But because someone is actually making a profit while serving low-income families, somehow the whole thing is ruined.  It’s as if social progress can only be made if taxpayers lose money.

It’s not accurate to say that Walmart is only able to provide low prices because it underpays its workers, who are themselves often low-income.  In fact, Walmart pays its workers above the industry average and offers health benefits rarely found in retail.  The reality is that Walmart primarily reduces prices by squeezing its suppliers.  Remember the prescription drug companies?  Ironically, anti-Walmart activists are really pro-Procter & Gamble.  Their chant should be “Charge poor people more for shampoo so that Procter & Gamble thrives!”  I guess that wouldn’t be a very good chant at a rally (I’d make a bad activist), but you get my point. 

Of course, the other groups that get squeezed are the unions.  But even if you believed that unions provided significant benefits to workers, we should all recognize that it would have to come at the expense of low-income consumers.  There is no free lunch.  And keep in mind that Walmart workers already receive above-industry-average wages and health benefits, so the additional benefits of unionization are more dubious.  Furthermore, outside of North America Walmart workers are mostly unionized (as are the workers of all of their major competitors in those markets) and the company still thrives. 

I know.  People will hold this post up as an example of how I’m somehow in the employ of Walmart.  Just to set the facts straight — I’m an employee of the University of Arkansas and am primarily paid by the taxpayers of Arkansas.  I’ve never heard anyone suggest that my (or anyone else’s)  receipt of money from the government presents a conflict of interest that disqualifies them from evaluating government programs.  I’m as free to criticize Arkansas policies as to criticize Walmart.  (And I do have criticisms of Walmart.  For example, the produce is lousy and the stores in Florida, when I lived there, looked dingy.)

It’s true that my department received a $20 million gift from which I draw some income.  But that $20 million endowment was initiated by an anonymous foundation (not connected to the Waltons) with a $10 million gift that was then matched by the University’s matching grant program, which applied to all gifts that met certain criteria.  It’s true that the matching grant money originally came from the estate of Sam Walton, but he passed away in 1992 and neither the Waltons nor Walmart control those dollars.  So, my connection to Walmart exists, but it is tenuous.  They certainly have no ability to control what I say or do.

But even if I were a corporate executive at Walmart, the issue is whether my argument is true, not with whom do I have a financial connection.  Walmart executives could make an argument and be right.  The intellectually honest way to exchange ideas is to address the merits of other people’s ideas, not analyze their motives for articulating those ideas. 

My assessment of the evidence is that Walmart really does help people save money and live better.  If you disagree, rebut the evidence.

11 Comments | politics | Tagged: , , , , | Permalink
Posted by Jay P. Greene

%d bloggers like this: