Jay P. Greene's Blog

In a classic piece of Flim-Flam, Amanda Ripley has a cover story in The Atlantic arguing that high school sports are bad for student achievement.  Her evidence?  She looked at Finland and South Korea, countries with higher test scores, and they put less emphasis on sports than do American schools.  They also eat a lot more fish in Finland and South Korea.  I don’t know how Ripley knows that de-emphasizing athletics is any more causally connected to higher achievement than fish consumption is.  But since flim-flam passes for evidence even in serious intellectual magazines, like The Atlantic, we have to endure this type of argument.

Happily, The Atlantic just ran a rejoinder from my former and current graduate students, Dan Bowen and Collin Hitt.  They actually reference evidence and properly consider questions of causation and conclude that high school athletics probably contribute to higher student achievement.  And they see no reason to believe that the absence of high school sports explains the difference between student achievement in the US and countries like Finland and South Korea.

But evidence, shmevidence — I’m not taking any chances.  So I’m urging school districts to increase serving fish in the cafeteria to replicate what the Finns and Koreans do and match their level of achievement.

I just noticed that according to the Blog Stats counter provided by WordPress and viewable in the right column of this website, this blog has now been viewed more than 1 million times.  Wow, do people have time to waste.

Of course, the WordPress counter does not capture nearly all of the total views, since I believe it does not include people who view this website through a reader or who see it re-posted in other places.  So I guess we really passed 1 million some time ago.

Still, it’s worth taking a moment to celebrate.  1,000,000.

It has been almost a decade since the publication of Education Myths.  At that time I was concerned that education policy was largely driven by emotional appeals and assumed facts.  “Don’t you love children?” and “we know education spending is plummeting” used to be regular arguments in policy discussions.  Ed Myths was a response to that type of policy argument and an attempt to demonstrate how systematic evidence could be used to inform policy-making.  I (along with co-authors Greg Forster and Marcus Winters) understood that the particular evidence we were citing would soon enough be out of date, but we hoped that our approach to using rigorous evidence could serve as a model for future policy debates.

Much progress has been made over the last nine years in the use of systematic evidence in education policy-making.  Mountains of data are being collected and scores of well-trained social scientists are applying cutting edge techniques and clever research designs to draw useful lessons from those data.  And policy-makers increasingly rely on this rigorous evidence when making decisions.  Emotional appeals and assumed facts have diminished in their influence.

Despite this progress, the field just can’t seem to shake the enduring attraction of the flim-flam man.  The flim-flam man resembles a social scientist and cites evidence to make his case, but often relies on faulty evidence drawn from flawed research designs as well as selective and distorted interpretations of evidence.  The flim-flam man uses the veneer of social science to disguise an agenda-driven or ill-conceived argument.

The most common manifestation of the flim-flam man in ed policy is the practitioner of “selection on dependent variable” arguments.  This is the person who says that Finland or South Korea or Massachusetts produce really great results, so we should do something that they do to make similar progress.  Of course, it is impossible to know from an examination of successful places why they are successful.  For all I know, Finland’s success is a function of the heavy concentration of reindeer, South Korea thrives because of delicious kimchee, and Massachusetts is blessed with excellent human capital to staff its schools and has had super folks like Sandra Stotsky and Bob Costrelll to direct its efforts.

Just because I choose to focus on Finland’s rigorous teacher preparation, South Korea’s emphasis on test-preparation over athletics, and Massachusett’s standards does not mean that those factors caused the success in question.  Identifying causation requires, at a minimum, observing that certain practices or policies tend to be present where there is success and absent where there is failure.  Only looking at these successful places tells us nothing about why they are successful because we do not know if those same practices or policies also existed in other, unsuccessful places.

The flaw of selection on dependent variable analyses is so obvious that it is shocking that education policy debates continue to be shaped by them.  I know that it is tempting to look at some place that is doing well and wonder what you could imitate to get the same results, but we just need to stop considering that evidence in education policy debates.

We should take a pledge — No more selection on dependent variable analyses!  No more divining the secrets of success in Finland!  Anyone who continues to present this kind of argument as evidence should be shunned as a flim-flam man.  A quack.

I take some heart from seeing that folks have started to critique Malcolm Gladwell as a flim-flam man.  Yes, he writes well.  Yes, he tells appealing stories.  But his reading of social science evidence is thin, selective, and often distorted.  In a review of Gladwell’s latest book in the Wall Street Journal, Christopher Chabris summarizes it well:

Mr. Gladwell has not changed his own strategy, despite serious criticism of his prior work. What he presents are mostly just intriguing possibilities and musings about human behavior, but what his publisher sells them as, and what his readers may incorrectly take them for, are lawful, causal rules that explain how the world really works. Mr. Gladwell should acknowledge when he is speculating or working with thin evidentiary soup. Yet far from abandoning his hand or even standing pat, Mr. Gladwell has doubled down.

If psychology is turning on Gladwell, then maybe education can turn on its own flim-flammers.

It is time once again for us to solicit nominations for the Al Copeland Humanitarian Award.  The criteria of the Al Copeland Humanitarian Award can be summarized by quoting our original blog post in which we sang the praises of Al Copeland and all that he did for humanity:

“Al Copeland may not have done the most to benefit humanity, but he certainly did more than many people who receive such awards.  Chicago gave Bill Ayers their Citizen of the Year award in 1997.  And the Nobel Peace Prize has too often gone to a motley crew including unrepentant terrorist, Yassir Arafat, and fictional autobiography writer, Rigoberta Menchu.   Local humanitarian awards tend to go to hack politicians or community activists.  From all these award recipients you might think that a humanitarian was someone who stopped throwing bombs… or who you hoped would picket, tax, regulate, or imprison someone else.

Al Copeland never threatened to bomb, picket, tax, regulate, or imprison anyone.  By that standard alone he would be much more of a humanitarian.  But Al Copeland did even more — he gave us spicy chicken.”

Last year’s winner of “The Al” was George P. Mitchell, a pioneer in the use of fracking to obtain more, cheap and clean natural gas.  As I wrote last year about why Mitchell won:

George P. Mitchell didn’t even invent the techniques that he commercialized to extract significantly more natural gas.  Mitchell’s efforts didn’t just reduce carbon emissions by making clean energy plentiful, as Matt documents in his nomination.  Mitchell demonstrated how improving the human condition, including improving the environment, is more likely to come from individual freedom and capitalism than from government coercion.

Yes, Mitchell was richly rewarded financially for his accomplishments, but we’ve already established that making money in no way undermines one’s case for having improved the human condition.

Mitchell won over a group of other worthy nominees:  Banksy, Ransom E. Olds, Stan Honey, and Alfred Fielding and Marc Chavannes.

In 2011 “The Al” went to Earle Haas, the inventor of the modern tampon.  Thanks to Anna for nominating him and recognizing that advances in equal opportunity for women had as much or more to do with entrepreneurs than government mandates.  Haas beat his fellow nominees:  Charles Montesquieu, the political philosopher, David Einhorn, the short-seller, and Steve Wynn, the casino mogul.

The 2010  winner of  “The Al” was Wim Nottroth, the man who resisted Rotterdam police efforts to destroy a mural that read “Thou Shall Not Kill” following the murder of Theo van Gogh by an Islamic extremist.  He beat out  The Most Interesting Man in the World, the fictional spokesman for Dos Equis and model of masculine virtue, Stan Honey, the inventor of the yellow first down line in TV football broadcasts, Herbert Dow, the founder of Dow Chemical and subverter of a German chemicals cartel, and Marion Donovan and Victor Mills, the developers of the disposable diaper.

And the 2009 winner of “The Al” was  Debrilla M. Ratchford, who significantly improved the human condition by inventing the rollerbag.  She beat out Steve Henson, who gave us ranch dressing,  Fasi Zaka, who ridiculed the Taliban,  Ralph Teetor, who invented cruise control, and Mary Quant, who popularized the miniskirt.

Nominations can be submitted by emailing a draft of a blog post advocating for your nominee.  If I like it, I will post it with your name attached.  Remember that the basic criteria is that we are looking for someone who significantly improved the human condition even if they made a profit in doing so.  Helping yourself does not nullify helping others.  And, like Al Copeland, nominees need not be perfect or widely recognized people.

The Research

The Education Next Article on the Effects of Field Trips to Crystal Bridges

The Education Next Article on the Long-Term Effects of Field Trips to the Walton Arts Center

Commentaries

New York Daily News

The Huffington Post

Art Museum Teaching

Radio

NPR Affiliate in Arkansas, “Ozarks at Large”

NPR Affiliate in Los Angeles, “Airtalk”

NPR Affiliate in New Hampshire, “Word of Mouth”

CBS Affiliate in San Francisco

TV:

KATV ABC

KFTA FOX

KFSM CBS

News Stories:

USA Today, State-by-State page (no link available)

The Atlantic

Education Week

Education Week (Walt Gardner)

Real Clear Arts

Politico

Deseret News

Tulsa World

The City Wire

Students First

District Administration

Podcasts

Gadfly Podcast (starting at 14:26)

Heartland Podcast

American RadioWorks

AP Coverage

Albany Times Union

Anchorage Daily News

Arkansas Online

Atlanta Journal Constitution

Austin American Statesman

Banner News

Belleville News Democrat

Beaufort Gazette

Beaumont Enterprise

Blytheville Courier News

Bradenton Herald

Centre Daily News

Dallas/Fort-Worth

Dayton Daily News

Fort Mill Times

Greenwich Time

Herald Online

Houston Chronicle

Idaho Statesman

Kansas City Star

Kentucky.com

KTBS

Ledger-Enquirer (Columbus, GA)

Miami Herald

Myrtle Beach

News Observer

San Antonio Express

San Francisco Chronicle

Seattle Post Intelligencer

Star Telegram

Sun Herald

Texarkana Gazette

The Bellingham Herald

The Daily Citizen

The Gazette (Colorado Springs)

The Modesto Bee

The News Times

The News Tribune

The Oklahoman

The Olathe News

The Olympian

The Republic (Columbus, IN)

The Sacramento Bee

The Telegraph

The Wichita Eagle

THV 11 CBS Little Rock

Tri-City Herald

Westport News

WRAL.com (North Carolina)

WSB TV Atlanta

(HT: Brian Kisida)

(UPDATED 10-1-13)

In case you missed it yesterday, you can view the event at Crystal Bridges at which we presented the results of our field trip study published in Education Next:

You can also listen to this interview that aired on the local NPR affiliates’s show, Ozarks at Large.

There’s an old joke about the social scientist who was searching for his keys at night under a lamp post.  His student came along to help and asked him where he last remembered having his keys.  He said he thought he dropped them further down the block.  “Well, why are you looking here?” asked the student.  “Because the light is better under the lamppost,” replied the social scientist.

This joke tells us a lot about education policy research.  Scores of researchers are slicing and dicing math and reading standardized test results in every way imaginable, and their policy recommendations are focused on how these outcomes can be maximized.  While important, those standardized test results don’t capture every outcome we expect from our education system.  We also expect our schools to prepare students to become civilized human beings by making them aware of our cultural heritage, teaching them to think critically, and instilling tolerant and empathetic values.  But because we don’t have readily available measures of these outcomes, education researchers generally don’t examine whether schools are successful in producing them.  We prefer to look only under the lamp post.

In a new study published today in Education Next, my colleagues Brian Kisida, Dan Bowen, and I look down the block from the lamp post.  We experimentally analyze the extent to which culturally enriching field trips to an art museum and a performing arts theater produce benefits for students.  We find that they do.  Students assigned by lottery to receive field trips learn academic content, increase critical thinking, become more tolerant and empathetic, and are more likely to become cultural consumers who seek these enriching experiences on their own in the future. In short, they become a little bit more civilized.

The benefits for disadvantaged students (minority, low-income, or rural) are generally two to three times larger than the average effect.  Schools appear to play a critical role in exposing disadvantaged students to culturally enriching experiences, which they may not get if schools do not take them.

Our main study focuses on a randomized controlled trial involving almost 11,000 students who were awarded field trips by lottery to see the Crystal Bridges Museum of American Art or to have their tour deferred.  We also have a brief sidebar summarizing a natural experiment in which attendance zone boundary changes caused about 1,300 students to experience more or fewer field trips to see live performances at the Walton Arts Center.  Both studies produce very consistent results.

The advantage of the Walton Arts Center study is that it shows that the benefits of culturally enriching field trips compound and endure over time.  But the identification of causation in the Crystal Bridges study is more airtight.  Together, they tell a powerful story about important educational outcomes that can be discovered when we look beyond the lamp post.

(Guest Post by Collin Hitt)

Charter schools push out students who fall behind academically – that’s how they achieve superior results, right?* This is one of the main claims of charter school opponents, a claim that calls the moral character of charter school employees into question.

Charter schools – especially “no excuses” schools – thrive off of promises to recruit and educate students who would otherwise be left behind. Poor kids. Minority kids. Kids with low test scores who can’t read. A movement that promises to educate these kids only to slyly turn them away in favor of their higher-performing peers, is a movement of charlatans.

A forthcoming article looks at the issue. Ron Zimmer and Cassandra Guarino,in Education Evaluation and Policy Analysis, examine the transfer-out patterns in a large urban school district with nearly 60 charter schools. It is by far the largest, high-quality study to ever look at the issue.

Their main finding:

The coefficient estimates suggest that low-performing students at TPS schools are 1% to 5% more likely (at statistically significant margin) to transfer than above-average students, although the statistical significance may be achieved in part due to the large sample size. Low-performing students are neither more nor less likely to transfer out of charter schools.

They find no evidence that charter schools are pushing out their lowest-performing students. But wait – did anybody else catch that? – what did they write about traditional public schools? From the next paragraph of their study (emphasis added):

This suggests that low-performing students are more likely to transfer out of a [traditional public school] than a charter school. But again, the differences are relatively small—about 5%. Overall, the results across all models provide no evidence that low-performing students are more likely to exit a charter school than a high-performing student or a low-performing student in a TPS.

This study shows that in one very large urban district, there’s no evidence that charter schools are pushing out their bottom performers. The same cannot be said of the district’s traditional public schools. Let’s be charitable to the traditional public schools in that district; perhaps the findings that their lower-performers are more likely to leave is a result of the fact that those parents are seeking to leave for charter schools. Or perhaps this finding is just a fluke with Zimmer and Guarino’s model, and there’s really no difference at all in the exit rates of low performers in the district’s charter and traditional schools. Those are plausible ways to interpret the findings. (Can anyone think of plausible, less charitable ways to view these findings?)

Now ask yourself, had Zimmer and Guarino instead arrived at the opposite results – charter schools are more likely than traditional public schools to see their lowest performing students leave – would charter opponents be this generous in their interpretation? My guess is that the web would be somewhat more atwitter about this study, which deserves better coverage than it’s getting.

* The most rigorous research on charter schools uses random assignment. In the intent-to-treat components of these studies, students are considered charter students if they’re ever offered a seat in a charter school. Regardless of whether the students accept the offer, and regardless of whether students transfer out of charter schools at a later date, they are still considered charter school students in the ITT analysis. Yet random assignment studies consistently find positive results for charter schools, especially in urban areas where they are most likely to encounter disadvantaged students. Even if charter schools were pushing out their students – which apparently they’re not doing, according Zimmer and Guarino – it couldn’t explain these gains.

The title of this post seems like the traditional zen koan asking “What is the sound of one hand clapping?”  How are we supposed to fix schools without fixing schools?  The answer to this question may not require Buddhist reflection.  We can fix schools — that is, traditional public schools — by going around them.  We can expand access to other educational options, including charter schools, voucher schools, tax-credit schools. ESAs, digital schooling, home-schooling, and hybrid schools.  We can also expand access to enriching non-school activities, like museums, theaters, historical sites, summer camps, and after-school programs.  Reformers should concentrate their energy on all of these non-traditional-school efforts and stop trying so hard to fix traditional public schools.

The main reason we should stop focusing on fixing traditional public schools is that, for the most part, they don’t want to be fixed.  The people who make their living off of those schools have reasons for wanting schools to be as they are and have enormous political resources to fend off efforts to fundamentally change things.  Trying to impose reforms like merit pay, centralized systems of teacher evaluation, new standards, new curriculum, new pedagogy, etc… on unwilling schools is largely a futile exercise.  They have the political resources to block, dilute, or co-opt these efforts in most instances.

Trying to impose these reforms despite fierce resistance from traditional public schools usually does not improve outcomes for students but it does produce a series of negative side-effects.  First, attempting to impose reforms on a politically powerful and unwilling school system generates an enormous amount of strife and hostility.  Teachers and their friends hate it.  Reformers waste energy and resources.  Little is changed but everyone walks away hurt, drained, and distracted from more productive activities.

Second, attempting to impose reforms on traditional public schools requires a significant increase in centralized political control.  Reformers can’t possibly fight their battles in thousands of individual school districts, so they favor centralizing power in the hands of big city mayors, state departments of education, and the federal government.  They see it as one-stop-shopping.  If they can cram their preferred reforms through those centralized authorities, then they think they will have won the battle in each district and school controlled by that centralized authority.   But they are likely to lose even when they can concentrate their fire on the centralized authority.  And even if they prevail at the centralized level, traditional public schools are usually able to subvert and render inert most reforms through poor implementation.  The reforms usually fail but the centralization remains, which is harmful in a variety of ways, such as generally undermining our long-standing and effective system of federalism and reducing access to educational alternatives through Tiebout choice.

Third, even in the rare cases where centralized reforms are adopted and implemented, the very nature of reforms that can jump those hurdles usually makes them ineffective or counter-productive.  Centralized reforms that can be adopted and implemented have to be watered-down enough to gain broad support for passage and implementation, rendering them mostly impotent.  And to the extent that they have some bite, they have to impose that bite uniformly on a large set of schools and circumstances, producing policies that are one size fits none.  Such reforms have to be crude things lacking in subtlety or nuance that could make them appropriate and effective in highly varied contexts.

Fourth, even if by some miracle an effective and appropriate centralized reform with bite is adopted and properly implemented, there is no natural political constituency to preserve the integrity of that reform over time.  These reforms may be adopted with support from business or taxpayer groups, but those political interests cannot sustain their focus on maintaining reforms over time.  They have to get back to their businesses and regular lives.  Meanwhile the angry teachers who had a reform crammed down their throats are still working in those schools and remain well-organized, ready to eviscerate reforms as soon as the temporarily-focused winning coalition moves on to other matters.  Centralized reforms to fix public schools do not create a constituency to protect them over time.  The coalition supporting centralized reforms is strongest at the moment of passage and steadily weakens over time, while opposing forces in traditional schools can bide their time and repeal or weaken reforms later.

The beauty of fixing schools by not fixing schools is that it generally avoids or reduces all of these problems.  Yes, traditional public schools resist the creation of alternatives, but they do not do so with the same ferocity that they oppose reforms that directly effect their daily working life.  Focusing on alternatives to traditional public schools also does not require any political centralization.  In fact, it generally encourages decentralized control over education.  Alternatives to traditional public schools do not impose one size fits none type solutions.  They let a thousand flowers bloom.  And alternatives to traditional schooling create their own political support that grows over time as more people benefit from those choice and non-school educational offerings.

I understand that urging reformers to focus on fixing traditional schools by not fixing traditional schools sounds like abandoning the millions of children who remain in those schools, but that is simply not the case.  The best hope for improving the situation of those children in traditional public schools is by expanding access to alternatives and enriching out-of-school experiences.  If we succeed in expanding access to quality alternatives, more and more of those children will benefit by being able to take advantage of those alternatives.  In addition, traditional public schools may be more willing and able to adopt reforms that are appropriate for their circumstances as they learn about what alternative providers are doing and feel some pressure to take steps to attract and retain their students.

Of course, expanding access to alternatives and improvements in the traditional system will likely be very gradual.  Some reformers are impatient and demand solutions now.  But there are no effective quick-fix reforms available.  It’s better to make gradual progress than inflict considerable damage in a rush to fix everything now.  And remember that just as starving children in Africa are not helped by our finishing all of the food on our plates, our futile efforts to impose centralized quick-fixes do not actually help those millions in traditional public schools.  The measure of a desirable reform should not be the extent to which it makes us feel like at least we are trying, even if those efforts are counter-productive.  We need to achieve the Buddhist serenity of fixing schools by not fixing schools.  Then we will understand what the sound of one hand clapping really is.

Last month I had a post, “Camp Liberty,” in which I wrote:

At the time my fellow counselors and I used to joke that “boys day”resulted in an anarchic state like Lord of the Flies, with the only exception being that we didn’t kill Piggy.  But looking back on it, I see that summer camp was probably the closest thing to true liberty that our kids had experienced.  It was certainly more conducive to liberty than school, which gave almost exclusive emphasis to obedience to authority.  School was where kids were trained to obey the state and become cogs in a giant corporate machine.  Camp was where they learned to be free….

I’ve argued before that schools might have a lot to learn from camps.  They are both engaged in the activity of trying to prepare young people for adult life.  But I think camps are much more effective at preparing young people to be free adults.  I even think camps are remarkably effective at conveying traditional academic content.  And they do so at much lower cost. 

Continuing this theme I want to draw your attention to a great essay by Mark Slouka in the Wall Street Journal a few weeks ago describing his memories of summer camp.  Be sure to follow the link and read the whole piece but here is a good snippet:

How do you describe bliss for a sixth-grade boy? We ate what we wanted, slept when we wanted. Nobody cared. There were ponds. Older girls, their hair shining in the afternoon sun, lay out naked on floating rafts. This was scary. And not….

And so it went, a blur of mud and glory. When problems appeared, solutions—both eloquent and effective—were right behind. After listening to me argue with a kid named Scotty Steinberg for a week, Don went to the barn and came back with two pairs of boxing gloves. We should “figure it out,” he said. Everybody made a circle around us (something boys are hard-wired to do) and Scotty and I banged away at each other. I won. I think. After that we were friends and talked about it a lot….

During our last week, Don drove us to a rock climbing place near New Paltz, N.Y., where he introduced us to a guy named Tray who knew about climbing. Tray, as I recall, was very strong and didn’t wear underwear, and his girlfriend, who lived with him in the tent next to ours, often didn’t wear anything at all. Every now and then the screen door would unzip and Tray or his girlfriend would emerge from a cloud of smoke that didn’t smell at all like my father’s cigarettes.

Did we climb a cliff and risk our lives? Why, yes, we did. When another camper named David Mosher and I proved we could do 10 pull-ups on a tree branch (this was the qualifying test), Tray escorted us up a 250-foot cliff. I can vouch for the fact that 250 feet is very high. The pine trees between my sneakers looked about an inch tall. David started to cry. I was too scared to cry. When we came down, though, we told everybody it was fun. And it was. By God, it was.

When my father picked me up at the end of that month, he hardly recognized the feral, grinning creature that gave him a quick kiss and crawled into the back seat of the car. Or maybe he did, having been 12 once himself.

Yep.  Camp Liberty.