Jay P. Greene's Blog

I’ve been railing against education policy arguments based on identifying places with successful outcomes and then claiming we should imitate certain practices or policy features of those places to achieve success elsewhere.  I argued forcefully against it in my review of Marc Tucker’s book.  Anytime someone says Finland (or Massachusetts or whoever) is doing well because they have high standards or little choice or no school athletics, or whatever, they are just engaging in quackery — pure BS.

It is a matter of basic logic that one cannot know what causes success only by looking at a successful place (or set of successful places).  You cannot know whether any factor contributes to success without also considering unsuccessful places and examining whether that same factor tends to be more present or absent in successful relative to unsuccessful places.  This is an error known as “selection on dependent variable” and it is taught in any decent introduction to research methods course.

People who regularly draw policy recommendations based on the Finland du jour should be made to hold up a giant sign that says “I do not understand the basic research methods of the field in which I claim to be an expert.”  And that’s just it.  These people aren’t really experts.  There used to be a time when a clever writer could be considered an expert on education policy simply by virtue of articulating a clear-sounding argument.  That time is gone.

Even journalists should be expected to have some basic understanding of the methods in education policy research.  No one would accept that science reporters could be ignorant of fundamental principles of the scientific method.  No one would accept that diplomatic correspondents would have no knowledge of diplomatic history.  Even journalists — especially when they are writing book-length arguments — need to have some understanding of research methods that would include the obvious point that no causal claims can be made from selection on dependent variable analyses.

So, I was delighted to see the Brookings Institution’s Russ Whitehurst take on David Kirp’s book, Improbable Scholars: The Rebirth of a Great American School System and a Strategy for America’s Schools, which draws lessons from an examination of one “successful” district — Union City, NJ.  Whitehurst begins by illustrating why selection on dependent variable does not allow for causal claims:

A while back, I read a journalistic account of a small island off Italy in which people live to be quite old.  Could it be because they sleep late, or drink lots of wine, or live a communal existence, or don’t eat refined sugar?  Unfortunately there is no way to know based on the information provided in the article if it is one or more of the lifestyle characteristics the author identified as distinctive, or whether something else is going on.  Even the claim of unusual longevity is questionable since there is no birth registry. And taking everything at face value, maybe there are other places in Italy in which people live as long or longer with different lifestyles.

Whitehurst goes on to question whether Union City is really such a successful district.  And then he applies the problems of his Italy example to Kirp’s use of Union City to make causal conclusions about how school improvement can be achieved.  Lastly, Whitehurst concludes with:

Once we have valid descriptions of the distinctive operational differences between good and not-so-good schools, controlling for differences in student background and out-of-school factors that are beyond district control, the social science of district reform can move to planned and carefully evaluated interventions.  That is our playbook. That is where we need to be.  I don’t think the path goes through Union City.

Amen, Brother Russ.

(Guest Post by Matthew Ladner)

The Friedman Foundation strikes again, with School Choice the Next Generation officers Butcher and Bedrick offering a new survey of parents in the Arizona Empowerment Scholarship Accounts program. Read the whole thing, but here is a useful summary graphic:

A number of years ago a Princeton philosophy professor, Harry Frankfurt, gave a brilliant lecture, “On Bullshit,”  which was later published as a very short book.  In the book Frankfurt spends some time defining the term, distinguishing it from similar concepts, like a lie or humbug.  He suggests that bullshit is something that is “grounded neither in a belief that it is true nor, as a lie must be, in a belief that it is not true. It is just this lack of connection to a concern with truth — this indifference to how things really are — that I regard as of the essence of bullshit.”

Frankfurt acknowledges that there are some positive uses of bullshit. It sometimes just allows us to get along.  Rather than struggle at all times with everyone over what the truth is, bullshit is something that we all spout and accept as — if you will excuse the imagery — a type of social lubricant.  We would never be able to get along in large organizations without a fair amount of bullshit, which is part of why we see so much of it in all of our work lives.  Politics, which requires managing conflict, is also a bullshit-laden activity.

While bullshit is unavoidable and sometimes useful, it is overall a very destructive thing.  As Frankfurt puts it, “bullshit is a greater enemy of the truth than lies are.”  It is corrosive to the very process by which we seek truth by embracing an indifference to the truth.  The triumph of bullshit is not the triumph of falsehood, but the triumph of nihilism.  Bullshit makes us not care about the truth, so why should we care about anything?  To maintain a good society, bullshit must be held in check.

Unfortunately, Frankfurt suspects that bullshit is growing, not being held in check.  It is growing, he suggests because bullshit is a part of communication and as communication grows so does bullshit.  We are talking all of the freakin’ time and on all matters of public and private concern.  The extent to which we are communicating about all of these things far exceeds our ability to know the truth of them or even be concerned with the process of discovering the truth about them.  Facebook and Twitter pre-date Frankfurt’s lecture, but they nicely illustrate the relationship between increasing communication and increasing bullshit.   Frankfurt also blames the rise of bullshit partially on post-modern philosophies that actively promote and rationalize an indifference to the truth.

But I suspect that there is another force at work in the growth of bullshit.  I suspect that as violence becomes less permissible within modern societies, bullshit has substituted for violence as a mechanism for manipulating or coercing others.  The famous Prussian military strategist, Carl von Clausewitz, once said “War is the continuation of policy by other means.”  I think bullshit is the continuation of violence by other means.

Earlier in our country’s history, before the domestic use of violence was so limited, differences were often settled through violence.  When Carnegie Steel was faced with a strike at a Pittsburgh plant, Henry Clay Frick called out 300 Pinkerton detectives who killed 16 striking workers and wounded 23 more.  Unions were similarly known to take baseball bats to workers who crossed their picket lines.

So when a modern day Andrew Carnegie, like Bill Gates, wants to have his way, he doesn’t hire a private army of Pinkerton detectives to beat us into submission.  Instead, he hires an army of foundation staff and advocacy organizations to spout bullshit.  And in response the unions don’t take up baseball bats, they take up blogging.  Violence, like bullshit, is indifferent to truth; it is simply a mechanism for prevailing.  As violence becomes less available as a strategy for winning a dispute, bullshitting becomes more prominent.

If bullshit is on the rise and is corrosive to a good society, how can we limit or even reduce it?  Science and its weaker sister, social science, are the antitheses of bullshit.  They are enterprises entirely committed to the pursuit of truth.  Unfortunately, bullshit has infiltrated science and social science as those activities become more politicized and embedded within large bureaucratic organizations.  The scientist or social scientist may be as likely to promote bullshit as to man the barricades against it.  Instead, we need something stronger, more resistance to corruption, than the scientist to fight bullshit.  We need the skeptic.

The skeptic is someone entirely devoted to the task of discovering and debunking bullshit.  The skeptic may be a scientist but often isn’t.  And the skeptic can often be mistaken about what is and is not bullshit.  But the skeptic is always on the prowl for bullshit and is even more committed to the process of finding truth than the mere scientist is.

Penn Jillette and Teller are worthy of “The Al” because they are the most active and effective skeptics of our era.  They are illusionists who have extended their professional interest in deceiving others for entertainment  into a professional interest in uncovering and debunking the deception of bullshit for entertainment.  For eight seasons they hosted a series on Showtime that was, appropriately enough, called Bullshit.  They targeted everything from alternative medicine to recycling to lie detectors to the Bible.  The have also crusaded (irony intended) against bullshit in their stage show, in magazine articles, and in TV appearances on other people’s shows.  Penn and Teller speak truth to bullshit.

As it is, we are already knee-deep in bullshit.  Were it not for the efforts of skeptics like Penn and Teller we might well need a life raft.  For this, they deserve “The Al.“

Marcus Winters has an excellent new study on charter schools and special education.  Why are there large gaps between the percentages of students classified as disabled in charter and traditional public schools?  A large part of the explanation — about 80% of the difference — can be explained by the fact that charters are just less likely to classify students as disabled and more likely to declassify them.  That is, charters have students with almost the same distribution of true disabilities as found in traditional public schools, they just don’t put labels on as many of them.  Here’s how Marcus put it:

The gap in special education rates between charter and traditional public schools grows considerably as students progress from kindergarten through third grade. A large part (80 percent) of the growth in this gap over time is that charter schools are less likely than district schools to classify students as in need of special education services and more likely to declassify them….

…the results do not suggest that charter schools are refusing to admit or are pushing out students with special needs. In fact, more students with previously identified disabilities enter charter schools than exit them as they progress through elementary grade levels…

By far, the most substantial growth in the special education gap occurs in the least severe category, that of specific learning disability. Rates of classification in what might be considered the more severe (and less subjective) categories of special education—autism, speech or language impairment, or intellectual disability—remain quite similar in charter and traditional public schools over time.

So… charter schools are not taking on students with dramatically different true disabilities; the traditional public schools are just more strongly inclined to classify the same kind of student as disabled.  And the traditional public schools mostly do this in the more subjective categories of disability, like specific learning disability.

These findings follow the same pattern as what Patrick Wolf, David Fleming, and John Witte discovered with special education and private schools participating in Milwaukee’s voucher program.  Schools of choice appear to be open to students with disabilities but aren’t as bureaucratically inclined to label students as disabled as are traditional public schools.

University of Arkansas graduate students, Anna Egalite and Jonathan Mills, have an excellent piece in Education Next on the effects of Louisiana’s voucher program on integration in schools.  This is an important empirical question because the US Department of Justice has filed suit against the state’s voucher program over concerns that it undermines federal desegregation efforts.

Egalite and Mills find that when students use vouchers to switch from a traditional public school to a private school, they tend to improve the racial integration of the public school they are leaving.  A transfer improves integration if the student’s departure would make that school more closely resemble the racial composition of the metropolitan area in which it is located.  So, if an African-American student leaves a school that is more heavily African-American than the broader community in which it is located, his or her transfer is positive for integration.  And the reverse is true.  Here are their statewide results:

When Egalite and Mills focus on the 34 school districts that are under federal supervision for desegregation in Louisiana, they find that the voucher program contributes to improved integration both in the public schools from which students are transferring and the private schools that they are entering.  Here are the results for those 34 districts:

The political boundaries of school districts and attendance zones appear to be an important impediment to integrating schools.  If we remove those boundaries by letting students mix voluntarily, we actually see more integration.    Maybe assuming that everyone is a racist and having the federal government try to force them not to be so racist is a less productive strategy than trying to remove barriers to voluntary and positive mixing of people from different backgrounds and different neighborhoods.

The U.S. Department of Justice filed their lawsuit without bothering to do this type of analysis.  It will be fascinating to hear how they react to this evidence, but since they are closed right now I guess we’ll just have to wait for their response.

In a classic piece of Flim-Flam, Amanda Ripley has a cover story in The Atlantic arguing that high school sports are bad for student achievement.  Her evidence?  She looked at Finland and South Korea, countries with higher test scores, and they put less emphasis on sports than do American schools.  They also eat a lot more fish in Finland and South Korea.  I don’t know how Ripley knows that de-emphasizing athletics is any more causally connected to higher achievement than fish consumption is.  But since flim-flam passes for evidence even in serious intellectual magazines, like The Atlantic, we have to endure this type of argument.

Happily, The Atlantic just ran a rejoinder from my former and current graduate students, Dan Bowen and Collin Hitt.  They actually reference evidence and properly consider questions of causation and conclude that high school athletics probably contribute to higher student achievement.  And they see no reason to believe that the absence of high school sports explains the difference between student achievement in the US and countries like Finland and South Korea.

But evidence, shmevidence — I’m not taking any chances.  So I’m urging school districts to increase serving fish in the cafeteria to replicate what the Finns and Koreans do and match their level of achievement.