Jay P. Greene's Blog

I’m teaching a seminar for the Honors College this semester on BS. It’s been a lot of fun and the students have been great.  Last week we were discussing the prevalence of BS in social science.  In particular, we were discussing the problems of file drawer and publication bias, p-hacking, and spurious relationships.  While considering why there is so much of this BS in social science we stumbled upon a possible explanation: perhaps there are just too many social scientists under too much pressure to regularly discover and report truths about human behavior when there just aren’t enough truths to be discovered.

Roughly estimating, there are at least 2,000 institutions worldwide that give priority to research and expect their faculty to produce it regularly.  And there are at least 50 active researchers in the social sciences at each of those institutions who depend on publishing novel insights about human beings, sometimes annually, in order to obtain and keep their jobs as well as receive promotions. In my back of the envelope calculation, there is demand for “discovering” roughly 100,000 true things about human behavior each year.

Now let’s consider the supply side.  My general worldview is that there is a very limited number of universally true things we could say about human behavior.  I’d wager that there are no more than several dozen true things that generally apply to human beings across time, place, and context.  And perhaps there are several hundred more contingently true things, observations that would be true for specified groups of people in particular circumstances.  The number of universally or contingently true observations we could make about human behavior may not exceed a thousand.

I understand that this description of the supply side is merely an assertion with which many may disagree.  But if you accept that much about human behavior is truly random or the function of idiosyncratic factors that make them impossible to predict, then you’d have to accept that the number of true observations about human behavior is quite limited. Even if it is more than a thousand, it is almost certainly well short of the number being demanded by social science researchers.

This shortage of truths to discover about human beings is especially severe when you consider that many of the possible true things have already been discovered.  Social science may only be about a century old, but the search for true observations about human beings goes as far back as we have written records.  Poets, philosophers, artists, and historians have been casting their nets for generalizations about human beings for millennia, so it is questionable how many fish are left in the sea.

I don’t want to be understood as making the equivalent of the claim that all discoveries have been made so let’s close the patent office.  I’m confident that there are new and interesting observations to be made about human behavior.  And I’m even more confident that we can do much to confirm or to dis-confirm previously made observations.  I just can’t escape the impression that the number of these true things to be said is dwarfed by the number of people with professional pressures to uncover them.

If I’m right, several implications follow.  The prevalence of BS in social science, including replication problems, file-drawer and publication bias, p-hacking, etc…, cannot be addressed with improved training or enforcement of more rigorous standards.  The pressure to make claims that are not really true is simply too great to be controlled by ethics or peers facing the same pressure.

Even if we could remove the external incentives for generating dubious findings by altering standards for promotion and tenure, we are left with the problem that searching for these findings is built into the self-definition of social scientists.  Social scientists think of themselves as explorers and they will continue to sail the world’s oceans shouting “Land!” at every mirage on the horizon even if much of the Earth has already been mapped.

I suspect that a main solution is to reduce the number of people engaged in social science research.  I don’t see this as particularly likely to happen given our desire to reproduce ourselves.  But pressure to reduce the number of social science researchers might come from those who foot the bill for these adventures, including taxpayers, tuition-paying students, and benefactors.

It might also be helpful if the social sciences begin to change how they think of what they do to be more like the humanities.  Scholars in the humanities are not so much focused on making new discoveries as they are on documenting and disseminating the insights of the past.  This makes them more interested in teaching and longer-form scholarship, like books.  Of course, given the celebrity-worship status hierarchy in the social sciences, how will we know who the coolest kids are if they do things that are hard to rank and compare, like teaching and only periodically writing books?

I don’t hold out much hope for the social sciences beginning to thin their ranks of researchers or shifting to a humanities orientation.  But thinking about the mismatch between how many people are searching for generalizations about human behavior and how many valid generalizations they are able to find is still useful for diagnosing how the social sciences may have gone astray.

(Guest post by Greg Forster)

OCPA carries my article on class sizes:

California enacted a big class-size reduction policy in 1996. It sounded easy when it was pitched to voters, but it ended up costing the state billions of dollars. And it produced no measurable improvement in any education outcomes—not test scores, not graduation rates, nothing.

Alas, the lesson was not learned. Florida enacted an even more ambitious class-size reduction policy in 2002. It cost the state $20 billion to implement as it was scaled up over eight years, and costs between $4 billion and $5 billion to maintain every year. And it produced no positive effect on education outcomes.

Let me know what you think!

(Guest post by Greg Forster)

Rand reports that a big randomized trial of super-lax discipline policies in Pittsburgh schools had negative effects on student achievement and big negative effects on African-American student achievement. Of course, Rand buries the negative finding under mountains of politically palatable pablum, but if you dig (search for “not all PERC impacts were positive”) you can find it just barely acknowledged. Or, if you grok geek, check out Table 6.7 on p. 56.

Via this thread, which reviews other studies with similar findings in LA and Philly. Backfill from Max Eden at (sigh) Fordham.

As I’ve written before on JPGB, the most destructive aspect here is that advocates have chosen to label the loosey-goosey discipline policy “restorative justice.” They accept a false and outrageous distinction between “restorative” and “retributive” justice, and cling to the absolutely unsupported prejudice that lax policies will be more restorative, while it is the desire for retributive justice that produces harsher policies. The reality is that in most cases, justice must be both retributive in intention and strict in application to be truly restorative. There is a place for mercy, but not too much and not too often. Nothing ruins young people more effectively, or prevents their “restoration” more completely, than lax discipline.

This is destructive because the lesson most people will take from their failures is that justice should not be restorative if we want it to be either retributive or effective:

The increasing tendency of some to dehumanize criminals and demand harsher and harsher treatment of them cannot be fought by advocacy of lax punishments in the name of “restorative justice.” It is directly caused by advocacy of lax punishments in the name of “restorative justice.”

Only retributive justice, which affirms that punishment is not an arbitrary tool of social control but a just and necessary consequence of the crime that the criminal is morally obligated to suffer, can be effective in restraining the abuse of criminals – and promoting their genuine restoration.

As C.S. Lewis once said, I plead for retributive justice not primarily for the sake of society, or for the sake of crime victims, but for the sake of the criminal.

When behavioral nudges using text messages became the flavor of the month a few years ago I expressed some serious reservations. In general, I was concerned that nudges substitute the preferences of distant experts for those of people who may understand their own situation better, thereby pushing people to do things against their better judgement.  These interventions may appear successful in the short run, especially when we examine near-term outcomes that are over-aligned with the intervention, but they may harm people over the long run.

In particular, I was responding to texting nudges being advocated by Ben Castleman and others to reduce “summer melt” by getting students to complete the FAFSA and enroll in college.  I wrote:

…even if sending text messages is successful at getting more low-income students to complete the FAFSA and enroll in college in the fall, it is unclear whether this ensures a positive outcome. Students who start college but then fail to finish may be hurt by forsaking employment and other training opportunities and taking on significant debt for a credential they never earn. The students who are accepted to college but then decide not to enroll may have just been deterred by an intimidating form, as Castleman suspects, or they may know things about themselves that made them rationally decide not to pursue a degree they are unlikely to complete. The 160-character solution may unwittingly push students into making decisions that are against their better judgment and end up harming them. Castleman has not reported retention and graduation rates from the texting intervention, so we do not know whether this behavioral nudge is helping or hurting students in the long term.

Well, Ben Castleman and Lindsay Page have finally released results on the longer-term effects of their “summer melt” texting nudge and they are pretty clearly negative.  That is, students randomly assigned to receive texts to remind them to complete the FAFSA while they are seniors in high school are significantly less likely to complete an AA or BA degree than those who were not nudged into completing the forms necessary to get financial aid and enroll in college.  As can be seen in Table 10, the treatment group was 1.7 percentage points less likely to complete a BA degree in 4 years (p< .01).  The treatment group was also .8 percentage points less likely to complete an AA degree after 2 years and 1.0 percentage point less likely to complete an AA after 3 years (p < .1).

Castleman and Page then focus on the subset of subjects who were in the uAspire program and for whom they had outcomes after 6 years.  At the end of 6 years the students who were randomly assigned to receive texting nudges were 2 percentage points less likely to have earned an AA degree (p < .1) and they were no more likely to have earned a BA degree. (See Table 12).

Castleman and Page also report results on whether students already enrolled in college are more likely to complete their degrees if they are randomly assigned to receive texts reminding them to renew the FAFSA so they can continue receiving financial assistance.  As shown in Table 18, students reminded to renew the FAFSA are no more likely to complete an AA or BA degree.

So, the longer-term results of these texting nudges are generally null or negative despite initially encouraging results that the intervention got more students to complete a form and enroll in college.  The problem is likely to be exactly what I suspected.  Students were being pushed into doing things that were against their own better judgement.

Researchers and foundation officials may think everyone should enroll in college but they don’t know each student’s circumstances and are very poorly positioned to know what is best for others.  Students and the advisors (family, counselors, educators, etc…) who know them and with whom they have authentic relationships understand the context better and are more likely to make good decisions.  Technocrats are inclined to manage things from afar, but this approach is very likely to end up hurting students.

The early, positive results for texting nudges received an enormous amount of attention, including an endorsement from Bill Gates and being featured on NPR’s Hidden Brain. Given that we now know that this type of nudge intervention may harm students, I hope there is a comparable amount of attention given to the release of the longer-term negative results.  There is no shame inherent in an intervention failing, but there are serious problems if we only tout temporary successes while ignoring long-run damages.

To their credit, Castleman and Page appear to be turning their attention in their newer research to “higher-touch” interventions that may cost more but may also have a better chance of providing guidance better suited to each student’s situation.  Higher-touch interventions also seem to acknowledge that success for students typically requires much more than a reminder or some information.  To succeed students need character traits that will help them make better decisions for themselves over and over as life presents an endless string of challenges.  To shape character requires human interaction and meaningful relationships.  That’s something that a “bot” or text message simply cannot do.

(Guest post by Greg Forster)

The Oklahoman carries my op-ed on why collective bargaining is a bad deal for teachers:

Teachers are like doctors and lawyers. Standardizing the work they do into a one-size-fits-all mold creates major headaches. But collective bargaining demands standardization, so processes and outputs can be negotiated.

The standardization demanded by collective bargaining is a major factor in all the complaints we’re accustomed to hearing from public school teachers — useless paperwork, unreasonable rules, rigid systems, dysfunctional bureaucracy. In a 2009 study of national data from the U.S. Department of Education, I compared public and private school teachers. The difference in teacher working conditions was dramatic.

Remember, you read it here first!

Rick Hess is releasing his EduScholar ranking tomorrow and we should expect a flurry of tweets and even university press releases bragging about people’s position on that measure. Even though Rick’s ranking (for which I serve as a nominating committee member) is a completely made up thing that has never been validated, people within our field act like it is incredibly important and care intensely how they are ranked.  I suspect that Rick knows this and that the ranking is epic-level trolling on his part.

But the field’s over-reaction to this unimportant ranking is a sign of its obsession with celebrity-worship. Even though almost no one outside of a few hundred people within our field cares about who we are or what we do, the folks within the field are crazed with a desire to have status and power in this universe that no one else even notices.  It’s as if we are in high school and everyone is obsessed with being accepted by the small circle of cool kids.  No one outside of your high school knows or cares who the cool kids are, but to those in the high school it feels like the most important thing in the world.  This is pretty much what education policy and other social sciences look like.

This excessive concern with status within our field is both the result of and contributor to a series of problems.  In economics, which seems the most afflicted with celebrity-worship, we see power and status concentrated in a small number of people within a small number of departments.  That small clique effectively controls the top journals in the field, dominates the main professional association, and has disproportionate influence over who is hired and tenured at those few departments.

Not surprisingly, this concentration of unchecked power leads to a variety of abusive behaviors.  People at the top of this status system can more easily maintain their power and help their friends, which is not only grossly unfair but also hinders a truly meritocratic pursuit of the best people, ideas, and research.  In addition, because those at the top are predominantly white and male, this strict status system excludes women, minorities, and all other newcomers who may differ from those with greater power.  And this concentration of unchecked power has also likely contributed to sexual harassment, intellectual theft, exploitation, and generally rude behavior.

Many people are beginning to speak out about these abusive behaviors.  There were several panels at the most recent ASSA meeting to document these issues and discuss what to do about them.  While this is all very encouraging, I fear that people may be missing what I suspect is the heart of the problem.  We can’t fix abusive and anti-intellectual behavior in social science by replacing a male-dominated status hierarchy with a more gender balanced system that still concentrates status and power so severely.  We suffer under a good old boy system, but we would continue to suffer even if they thought they were good, were women, and much younger.  The problem is the unchecked concentration of power and status.

I think we would be much better off if control over the main journals and professional associations was dispersed outside of a small number of people at a small number of institutions.  Boards for journals and professional associations tend to be self-replicating bodies that draw from the same incestuous pool.  They should consider adopting by-laws or at least norms that push them to consider finding new members outside of their familiar, friend and colleague circles.  It would also be helpful for professional associations and journals to adopt real grievance procedures so that intellectually dishonest or personally abusive behavior could be considered with due process and treated with appropriate sanctions.  My personal experience is that this is not happening.

But even more important than changing association and journal rules and procedures, we need to abandon the culture of celebrity worship.  No one in education policy or other social sciences is actually that important.  At most they are King of the Lilliputians.  At worst they are folks who were excluded from the in-groups in high school now taking their revenge by terrorizing those beneath them.  For the most part, no one in the outside world cares about who we are, what journals we publish in, what rankings we get, etc…  None of us are celebrities.

People at the top of our status system only have power because we have given it to them by acting like they are celebrities and that their position really matters.  The solution is the same as when we were in high school.  The only way to avoid being terrorized by the in-group is to stop caring about the in-group.  They just don’t matter.  Form your own chess club, play D&D, and ignore the football team and cheerleaders.

So when Rick’s ranking comes out, have a good laugh and think about how much those who are striving to be at the top of some silly list are wasting their lives. When high school is finished no one will remember or care that they were once really cool.