I’m teaching a seminar for the Honors College this semester on BS. It’s been a lot of fun and the students have been great. Last week we were discussing the prevalence of BS in social science. In particular, we were discussing the problems of file drawer and publication bias, p-hacking, and spurious relationships. While considering why there is so much of this BS in social science we stumbled upon a possible explanation: perhaps there are just too many social scientists under too much pressure to regularly discover and report truths about human behavior when there just aren’t enough truths to be discovered.
Roughly estimating, there are at least 2,000 institutions worldwide that give priority to research and expect their faculty to produce it regularly. And there are at least 50 active researchers in the social sciences at each of those institutions who depend on publishing novel insights about human beings, sometimes annually, in order to obtain and keep their jobs as well as receive promotions. In my back of the envelope calculation, there is demand for “discovering” roughly 100,000 true things about human behavior each year.
Now let’s consider the supply side. My general worldview is that there is a very limited number of universally true things we could say about human behavior. I’d wager that there are no more than several dozen true things that generally apply to human beings across time, place, and context. And perhaps there are several hundred more contingently true things, observations that would be true for specified groups of people in particular circumstances. The number of universally or contingently true observations we could make about human behavior may not exceed a thousand.
I understand that this description of the supply side is merely an assertion with which many may disagree. But if you accept that much about human behavior is truly random or the function of idiosyncratic factors that make them impossible to predict, then you’d have to accept that the number of true observations about human behavior is quite limited. Even if it is more than a thousand, it is almost certainly well short of the number being demanded by social science researchers.
This shortage of truths to discover about human beings is especially severe when you consider that many of the possible true things have already been discovered. Social science may only be about a century old, but the search for true observations about human beings goes as far back as we have written records. Poets, philosophers, artists, and historians have been casting their nets for generalizations about human beings for millennia, so it is questionable how many fish are left in the sea.
I don’t want to be understood as making the equivalent of the claim that all discoveries have been made so let’s close the patent office. I’m confident that there are new and interesting observations to be made about human behavior. And I’m even more confident that we can do much to confirm or to dis-confirm previously made observations. I just can’t escape the impression that the number of these true things to be said is dwarfed by the number of people with professional pressures to uncover them.
If I’m right, several implications follow. The prevalence of BS in social science, including replication problems, file-drawer and publication bias, p-hacking, etc…, cannot be addressed with improved training or enforcement of more rigorous standards. The pressure to make claims that are not really true is simply too great to be controlled by ethics or peers facing the same pressure.
Even if we could remove the external incentives for generating dubious findings by altering standards for promotion and tenure, we are left with the problem that searching for these findings is built into the self-definition of social scientists. Social scientists think of themselves as explorers and they will continue to sail the world’s oceans shouting “Land!” at every mirage on the horizon even if much of the Earth has already been mapped.
I suspect that a main solution is to reduce the number of people engaged in social science research. I don’t see this as particularly likely to happen given our desire to reproduce ourselves. But pressure to reduce the number of social science researchers might come from those who foot the bill for these adventures, including taxpayers, tuition-paying students, and benefactors.
It might also be helpful if the social sciences begin to change how they think of what they do to be more like the humanities. Scholars in the humanities are not so much focused on making new discoveries as they are on documenting and disseminating the insights of the past. This makes them more interested in teaching and longer-form scholarship, like books. Of course, given the celebrity-worship status hierarchy in the social sciences, how will we know who the coolest kids are if they do things that are hard to rank and compare, like teaching and only periodically writing books?
I don’t hold out much hope for the social sciences beginning to thin their ranks of researchers or shifting to a humanities orientation. But thinking about the mismatch between how many people are searching for generalizations about human behavior and how many valid generalizations they are able to find is still useful for diagnosing how the social sciences may have gone astray.
Jay P. Greene's Blog 
This is a great post! Ironically, the “too many researchers” point has been made in other fields: https://blogs.scientificamerican.com/observations/are-scientists-doing-too-much-research/ I understand, of course, that as a social scientist with a blog, you feel pressure to say something new in every blog post. 🙂
That’s fantastic!
To be clear, I am not really troubled by repeating claims that have been previously made since confirmation can be helpful. More troubling is the pressure to yell “Land!” at the sight of every mirage on the horizon.
Excellent post—there are only really a handful of good scholars, researchers doing REAL good empirical research- and they are often hampered by lack of funds, lack of support and time constraints. So, what we get is a lot of small sample size research, we get a lot of short term and very little long term research with long term follow up and a lot of minuscule studies focusing on so small a group that the results are only applicable to a very small number of individuals. So, we have literature reviews that while good, add nothing new really to our knowledge base, and a plethora of these open access journals that are charging exorbitant amounts of money ( APC=Article Processing Charge ) to get published. While I also support the Humanities, there should be a place also for good experimental and empirical research that is data based and has some real impact. On the other hand, with the increasing heterogeneity in the schools, we also have to be careful of results and generalizations. But, Professor Green has written a good, important article which should be read and discussed !
I wonder if your estimate of working researchers is too high – smaller institutions will not have 50 social scientists each – but I’m too lazy to check the DES.
A lot depends on what counts as a truth worth discovering. There are plenty of questions social science could worthily devote its labor power to, but they tend to be 1) useful but unglamorous follow-up investigations of secondary and tertiary implications of “big” findings; 2) yet more useful but yet less glamorous ongoing monitoring of basic data trends (when you first did real grad rates it was huge, but who’s tracking those rates annually now?); or 3) even yet more useful but even yet less glamorous questioning of basic assumptions about social behavior that are too politically sensitive for us to permit much serious investigation (eg sociology of religion).
The problem is not that there’s nothing worth discovering, the problem is that in a cultural environment where “science” is the only public moral authority, social science is expected to serve as a substitute for the public role formerly played by the priesthood. The real science is either too unglamorous or too socially upsetting for it to play that role.
We may be saying similar things in different ways. Religion, like the humanities, may have already observed many of the generally true things we might say about human behavior. But we don’t accept those observations as valid anymore because Science is the only form of knowledge that we now readily accept.
But we may not agree as much on how many contingently true things there are to say. So much of human behavior is random or idiosyncratic that it cannot be predicted with scientific confidence.
We’re not far apart but I want to add a subtle development of the principle. In both science and theology, I think we need to distinguish between primary and secondary truths. The primary truths – or to be more precise, the subset of all possible primary truths that are currently discoverable (for both science and theology develop historically and can only discover “the next step” at any given time) – are few and largely already discovered. Secondary, tertiary (and so on) truths are always discoverable in abundance. But we don’t reward social scientists for their work on those because we want the wrong things from social science. By contrast, natural science and theology fully understand their own limitations and have developed systems that adequately reward those who develop secondary and tertiary discoveries (although obviously much greater honors await the few primary discoverers).
To the extent that we do differ, I think you are operating more on the expectation of a small, relatively static set of discoverable primary truths, whereas I’m assuming ongoing development and new discovery is always in possible in principle.
Very interesting post, and I think an important insight. But I disagree on the number of important, continently true things out there for social science to discover. They tend to be practically oriented, specific to a cultural or political landscape at a particular moment.
We conduct dozens of RCTs looking at how messages move voters on policy or candidate support, and we should expect that what we find changes across time and context. Think about how voters respond to the race or sex of a candidate … it’s certainly not the same now as it was in 1970, 1980, or 2000. Nor is it the same for Republican and Democrats, or Mormons and the non-religious, or in a primary vs general election.
I think there are a huge number of these types of contingent, but important questions to answer. Just a “for instance,” this relatively new use of conjoint experiments to illuminate various contingent human responses:
https://onlinelibrary.wiley.com/doi/abs/10.1111/ajps.12138
Thanks! But if our discoveries become too contingent, they become indistinguishable from idiosyncratic. That is, we can never really know whether the particular circumstances that produced that finding would apply to other circumstances. We can’t predict and replicate as science would require.
That all depends on how wide or narrow the relevant field of assumptions is. Most of the actual history of the cosmos is not considered a contingency for purposes of social science, but a given. If you’re not prepared to say that the Big Bang itself and every discrete event since then is a relevant contingency, then we’ve already established what kind of social scientist you are and now we’re just haggling over price.
An excellent post.
Is there any sort of article that lays out “Here are the top few things that social science has produced in the last, say, 10 years?”
I.e., if those things have huge social value, helps us digest the value of the 100k researchers.
In cancer:
2017 first FDA approved gene therapy, may one day replace chemo
2016 first bladder cancer new drug in 30 years
2015 3 immunotherapies for lung + CDK inhibitors for breast
Etc.
I’m not aware of a list like that, but then again I’m not really a good PR flak for social science. Check with APPAM, AERA, and AEFP to see if they know of anything given that they are the main professional associations in education policy research. You could also try similar orgs for economics, psychology, political science, and sociology in addition to more specialty orgs for criminal justice, urban planning, etc… My guess is you won’t find what you are looking for.
[…] To read the article, click here. […]
Good post. I suspect that is right. Perhaps the recent efforts at replication will at least nudge the field towards fewer, better studies. But if most papers are co-authored with dozens, that will also necessitate a change in promotion.
Related from the humanities: in the 1980s one of my undergrad professors, Michael Aeschlimann (humanities), argued that the dissertation requirements “new, true, and different” led to the neglect of the vast amount of true that is not new. More recently, and in line with your post, Tim van Gelder argued there were about twice as many philosophers as there should be. Answering his own implicit challenge, he left academic philosophy to pursue applied critical thinking.