Jay P. Greene's Blog

It’s been a busy week with the publication of an op-ed by me and Rick Hess in the Wall Street Journal and a study in Education Next documenting the monolithic partisan composition of education reform advocates and those who conduct research on those efforts. One might think that folks committed to evidence-based decision-making would be very interested in facts about their field, but their social media response has generally been counter-productive and fact-free.  Those responses have focused on how they are not to blame, how Republicans are icky anyway, and how many of their best friends are Republicans.  I’m not bothering to link to those responses because there really is no point.  If folks are happy with a uniformly Democratic movement, then they are welcome to keep it… as long as someone continues to be willing to pay for this party.  Given the groupthink and political ineffectiveness that is likely to result from this lack of heterodoxy, I can only wonder why and for how long funders will subsidize it.

Lost in the shuffle of this busy week, some graduate students and I released two new studies of the medium-term effects of students receiving multiple arts-focused field trips to the Woodruff Arts Center in Atlanta.  We randomly assigned school groups to a treatment that involved three field trips per year to visit an art museum, see live theater, and listen to the symphony, or to a control condition.  Among the treated students, some received 3 experiences over 1 year and some received 6 experiences over 2 years.

We split the analyses into two separate reports.  The first, led by Heidi Holmes Erickson, found that these arts-focused field trips improved school engagement, as measured by disciplinary infractions and survey responses, as well as increased standardized test scores in math and reading. These benefits persisted even one year after treatment ended for the first cohort in the study.

The second study, led by Angela Watson, examined social-emotional outcomes.  It found that exposure to multiple arts-focused field trips increased social perspective taking and tolerance.  It also found evidence of an improvement among treated female students in their conscientiousness, as measured by survey effort.

Heidi and Angela will be presenting these results at the Association for Education Finance and Policy conference next week.  Please attend their sessions to learn more about this research and to provide suggestions for improving their papers.  And if folks at AEFP are also interested in engaging in a productive discussion of how to improve the intellectual and ideological diversity of the organization, that would also be wonderful.

I’m teaching a seminar for the Honors College this semester on BS. It’s been a lot of fun and the students have been great.  Last week we were discussing the prevalence of BS in social science.  In particular, we were discussing the problems of file drawer and publication bias, p-hacking, and spurious relationships.  While considering why there is so much of this BS in social science we stumbled upon a possible explanation: perhaps there are just too many social scientists under too much pressure to regularly discover and report truths about human behavior when there just aren’t enough truths to be discovered.

Roughly estimating, there are at least 2,000 institutions worldwide that give priority to research and expect their faculty to produce it regularly.  And there are at least 50 active researchers in the social sciences at each of those institutions who depend on publishing novel insights about human beings, sometimes annually, in order to obtain and keep their jobs as well as receive promotions. In my back of the envelope calculation, there is demand for “discovering” roughly 100,000 true things about human behavior each year.

Now let’s consider the supply side.  My general worldview is that there is a very limited number of universally true things we could say about human behavior.  I’d wager that there are no more than several dozen true things that generally apply to human beings across time, place, and context.  And perhaps there are several hundred more contingently true things, observations that would be true for specified groups of people in particular circumstances.  The number of universally or contingently true observations we could make about human behavior may not exceed a thousand.

I understand that this description of the supply side is merely an assertion with which many may disagree.  But if you accept that much about human behavior is truly random or the function of idiosyncratic factors that make them impossible to predict, then you’d have to accept that the number of true observations about human behavior is quite limited. Even if it is more than a thousand, it is almost certainly well short of the number being demanded by social science researchers.

This shortage of truths to discover about human beings is especially severe when you consider that many of the possible true things have already been discovered.  Social science may only be about a century old, but the search for true observations about human beings goes as far back as we have written records.  Poets, philosophers, artists, and historians have been casting their nets for generalizations about human beings for millennia, so it is questionable how many fish are left in the sea.

I don’t want to be understood as making the equivalent of the claim that all discoveries have been made so let’s close the patent office.  I’m confident that there are new and interesting observations to be made about human behavior.  And I’m even more confident that we can do much to confirm or to dis-confirm previously made observations.  I just can’t escape the impression that the number of these true things to be said is dwarfed by the number of people with professional pressures to uncover them.

If I’m right, several implications follow.  The prevalence of BS in social science, including replication problems, file-drawer and publication bias, p-hacking, etc…, cannot be addressed with improved training or enforcement of more rigorous standards.  The pressure to make claims that are not really true is simply too great to be controlled by ethics or peers facing the same pressure.

Even if we could remove the external incentives for generating dubious findings by altering standards for promotion and tenure, we are left with the problem that searching for these findings is built into the self-definition of social scientists.  Social scientists think of themselves as explorers and they will continue to sail the world’s oceans shouting “Land!” at every mirage on the horizon even if much of the Earth has already been mapped.

I suspect that a main solution is to reduce the number of people engaged in social science research.  I don’t see this as particularly likely to happen given our desire to reproduce ourselves.  But pressure to reduce the number of social science researchers might come from those who foot the bill for these adventures, including taxpayers, tuition-paying students, and benefactors.

It might also be helpful if the social sciences begin to change how they think of what they do to be more like the humanities.  Scholars in the humanities are not so much focused on making new discoveries as they are on documenting and disseminating the insights of the past.  This makes them more interested in teaching and longer-form scholarship, like books.  Of course, given the celebrity-worship status hierarchy in the social sciences, how will we know who the coolest kids are if they do things that are hard to rank and compare, like teaching and only periodically writing books?

I don’t hold out much hope for the social sciences beginning to thin their ranks of researchers or shifting to a humanities orientation.  But thinking about the mismatch between how many people are searching for generalizations about human behavior and how many valid generalizations they are able to find is still useful for diagnosing how the social sciences may have gone astray.

When behavioral nudges using text messages became the flavor of the month a few years ago I expressed some serious reservations. In general, I was concerned that nudges substitute the preferences of distant experts for those of people who may understand their own situation better, thereby pushing people to do things against their better judgement.  These interventions may appear successful in the short run, especially when we examine near-term outcomes that are over-aligned with the intervention, but they may harm people over the long run.

In particular, I was responding to texting nudges being advocated by Ben Castleman and others to reduce “summer melt” by getting students to complete the FAFSA and enroll in college.  I wrote:

…even if sending text messages is successful at getting more low-income students to complete the FAFSA and enroll in college in the fall, it is unclear whether this ensures a positive outcome. Students who start college but then fail to finish may be hurt by forsaking employment and other training opportunities and taking on significant debt for a credential they never earn. The students who are accepted to college but then decide not to enroll may have just been deterred by an intimidating form, as Castleman suspects, or they may know things about themselves that made them rationally decide not to pursue a degree they are unlikely to complete. The 160-character solution may unwittingly push students into making decisions that are against their better judgment and end up harming them. Castleman has not reported retention and graduation rates from the texting intervention, so we do not know whether this behavioral nudge is helping or hurting students in the long term.

Well, Ben Castleman and Lindsay Page have finally released results on the longer-term effects of their “summer melt” texting nudge and they are pretty clearly negative.  That is, students randomly assigned to receive texts to remind them to complete the FAFSA while they are seniors in high school are significantly less likely to complete an AA or BA degree than those who were not nudged into completing the forms necessary to get financial aid and enroll in college.  As can be seen in Table 10, the treatment group was 1.7 percentage points less likely to complete a BA degree in 4 years (p< .01).  The treatment group was also .8 percentage points less likely to complete an AA degree after 2 years and 1.0 percentage point less likely to complete an AA after 3 years (p < .1).

Castleman and Page then focus on the subset of subjects who were in the uAspire program and for whom they had outcomes after 6 years.  At the end of 6 years the students who were randomly assigned to receive texting nudges were 2 percentage points less likely to have earned an AA degree (p < .1) and they were no more likely to have earned a BA degree. (See Table 12).

Castleman and Page also report results on whether students already enrolled in college are more likely to complete their degrees if they are randomly assigned to receive texts reminding them to renew the FAFSA so they can continue receiving financial assistance.  As shown in Table 18, students reminded to renew the FAFSA are no more likely to complete an AA or BA degree.

So, the longer-term results of these texting nudges are generally null or negative despite initially encouraging results that the intervention got more students to complete a form and enroll in college.  The problem is likely to be exactly what I suspected.  Students were being pushed into doing things that were against their own better judgement.

Researchers and foundation officials may think everyone should enroll in college but they don’t know each student’s circumstances and are very poorly positioned to know what is best for others.  Students and the advisors (family, counselors, educators, etc…) who know them and with whom they have authentic relationships understand the context better and are more likely to make good decisions.  Technocrats are inclined to manage things from afar, but this approach is very likely to end up hurting students.

The early, positive results for texting nudges received an enormous amount of attention, including an endorsement from Bill Gates and being featured on NPR’s Hidden Brain. Given that we now know that this type of nudge intervention may harm students, I hope there is a comparable amount of attention given to the release of the longer-term negative results.  There is no shame inherent in an intervention failing, but there are serious problems if we only tout temporary successes while ignoring long-run damages.

To their credit, Castleman and Page appear to be turning their attention in their newer research to “higher-touch” interventions that may cost more but may also have a better chance of providing guidance better suited to each student’s situation.  Higher-touch interventions also seem to acknowledge that success for students typically requires much more than a reminder or some information.  To succeed students need character traits that will help them make better decisions for themselves over and over as life presents an endless string of challenges.  To shape character requires human interaction and meaningful relationships.  That’s something that a “bot” or text message simply cannot do.

Rick Hess is releasing his EduScholar ranking tomorrow and we should expect a flurry of tweets and even university press releases bragging about people’s position on that measure. Even though Rick’s ranking (for which I serve as a nominating committee member) is a completely made up thing that has never been validated, people within our field act like it is incredibly important and care intensely how they are ranked.  I suspect that Rick knows this and that the ranking is epic-level trolling on his part.

But the field’s over-reaction to this unimportant ranking is a sign of its obsession with celebrity-worship. Even though almost no one outside of a few hundred people within our field cares about who we are or what we do, the folks within the field are crazed with a desire to have status and power in this universe that no one else even notices.  It’s as if we are in high school and everyone is obsessed with being accepted by the small circle of cool kids.  No one outside of your high school knows or cares who the cool kids are, but to those in the high school it feels like the most important thing in the world.  This is pretty much what education policy and other social sciences look like.

This excessive concern with status within our field is both the result of and contributor to a series of problems.  In economics, which seems the most afflicted with celebrity-worship, we see power and status concentrated in a small number of people within a small number of departments.  That small clique effectively controls the top journals in the field, dominates the main professional association, and has disproportionate influence over who is hired and tenured at those few departments.

Not surprisingly, this concentration of unchecked power leads to a variety of abusive behaviors.  People at the top of this status system can more easily maintain their power and help their friends, which is not only grossly unfair but also hinders a truly meritocratic pursuit of the best people, ideas, and research.  In addition, because those at the top are predominantly white and male, this strict status system excludes women, minorities, and all other newcomers who may differ from those with greater power.  And this concentration of unchecked power has also likely contributed to sexual harassment, intellectual theft, exploitation, and generally rude behavior.

Many people are beginning to speak out about these abusive behaviors.  There were several panels at the most recent ASSA meeting to document these issues and discuss what to do about them.  While this is all very encouraging, I fear that people may be missing what I suspect is the heart of the problem.  We can’t fix abusive and anti-intellectual behavior in social science by replacing a male-dominated status hierarchy with a more gender balanced system that still concentrates status and power so severely.  We suffer under a good old boy system, but we would continue to suffer even if they thought they were good, were women, and much younger.  The problem is the unchecked concentration of power and status.

I think we would be much better off if control over the main journals and professional associations was dispersed outside of a small number of people at a small number of institutions.  Boards for journals and professional associations tend to be self-replicating bodies that draw from the same incestuous pool.  They should consider adopting by-laws or at least norms that push them to consider finding new members outside of their familiar, friend and colleague circles.  It would also be helpful for professional associations and journals to adopt real grievance procedures so that intellectually dishonest or personally abusive behavior could be considered with due process and treated with appropriate sanctions.  My personal experience is that this is not happening.

But even more important than changing association and journal rules and procedures, we need to abandon the culture of celebrity worship.  No one in education policy or other social sciences is actually that important.  At most they are King of the Lilliputians.  At worst they are folks who were excluded from the in-groups in high school now taking their revenge by terrorizing those beneath them.  For the most part, no one in the outside world cares about who we are, what journals we publish in, what rankings we get, etc…  None of us are celebrities.

People at the top of our status system only have power because we have given it to them by acting like they are celebrities and that their position really matters.  The solution is the same as when we were in high school.  The only way to avoid being terrorized by the in-group is to stop caring about the in-group.  They just don’t matter.  Form your own chess club, play D&D, and ignore the football team and cheerleaders.

So when Rick’s ranking comes out, have a good laugh and think about how much those who are striving to be at the top of some silly list are wasting their lives. When high school is finished no one will remember or care that they were once really cool.

The blog has been a little empty lately and your shopping carts may be too full, so I thought I would share some songs you might like that you’ve probably never heard before.  Consider it my gift to you.

First up, we have this beautiful song by the Vulgar Boatmen.  I wrote a blog post before about how great this band is (was), but I didn’t mention this gem.  It’s called There’s a Family.  Here is the studio version:

Here is a live version from a club concert in the early 90s.  I’m not sure which version I enjoy more.

Next we have the “Twee” band Allo Darlin’.  It’s probably that I’m getting old, but I don’t mind a sweet pop song, especially these vulnerable and heartfelt pieces:

You’ve probably heard the Kinks’ song, Strangers, but I bet you haven’t heard this cover by Lucius before:

This Tiny Desk Concert by Lucius is also pretty amazing.  I especially enjoy around the 12 minute mark when they are asked if they would play one more song and they then scavenge through the desks to find items to use for percussion while playing Genevieve.

These aren’t quite Matt’s punk or heavy metal covers, but I hope you enjoy them anyway.

It was a very crowded field of excellent nominees for this year’s Al Copeland Humanitarian Award. In total there were 8 nominees (two of whom shared the honor): Leo Moracchioli, Richard Garfield, Elizabeth Vandiver, Eric Lundgren, Adam Butler and Autumn Thomasson, George Henry Thomas, and Joy Morton.

Of all of these worthy individuals, Joy Morton best exemplifies the way in which Al Copeland improved the human condition.  Morton, like Copeland, promoted good by doing well.  As Collin noted in his post, Morton sought a competitive advantage for his salt company by adding iodine and advertising the health benefits of doing so.  It was known at the time that small amounts of iodine could prevent goiters, which were a widespread and damaging problem throughout America’s heartland. But no one was doing anything about this until Morton saw a way to make money from adding iodine to people’s diet.

It was later learned that iodine is crucial to healthy brain development.  By adding iodine to salt, Morton reduced cognitive disabilities among those with the lowest access to iodine in their diet, raising IQs by one full standard deviation in that population. Collin emphasized how much good Morton achieved through his profit-seeking enterprise relative to what has been achieved by billions in non-profit expenditures:

One. Standard. Deviation. Countless foundations have invested countless dollars to achieve impacts a fraction of that size in [a] tiny fraction of the population – and most have failed. Morton accomplished it all with table salt.

Al Copeland similarly improved the human condition through a profit-seeking enterprise.  Rather than prevent goiters and raise IQs, Copeland satisfied our desire for spicy chicken.  And both efforts have in common a significant reliance on salt.

Leo Moracchioli shares with Morton and Copeland the fact that he makes money from his humanitarian activities.  Making heavy metal covers on Youtube brings plenty of joy to his followers as well as money to his pocket.  And Matt was right to note the importance of “disintermediation” in producing this and other positive developments.  But it is hard for fun music to compare with preventing goiters and raising IQs let alone to providing spicy chicken.

Ben Ladner’s personal and well-written nomination of Magic: The Gathering’s creator, Richard Garfield  was also compelling.  But like my previous nomination of D&D promoter, Gary Gygax, Garfield falls short.  As much as I identify with and root for the Geek tribe, their amusement and acts of solidarity do not rise to the level of improving the human condition like spicy chicken does.

My nomination of Elizabeth Vandiver also falls short.  Promoting awareness of human nature through understanding of Classical Mythology is enormously important work, but Vandiver reaches too few people to make enough of a difference.  If only our schools thought this was an important part of their job and made use of Vandiver’s materials, it might be a different story.

Greg had several nominees.  We may have to consider a rule regarding whether an individual can have multiple nominees in a single year and whether multiple people can share a nomination.  In any event, Greg’s nomination of Eric Lundgren was excellent but it felt more like a Higgy nomination for Bill Gates. Making use of old computer parts is indeed noble, but the way Microsoft sought to block it shows that profit-seeking enterprises can also promote bad while doing well.  The nomination of Adam Butler and Autumn Thomasson for providing legal assistance to lemonade stands while also making a profit selling lemonade also sounds like a Higgy nomination for the PLDDs who seek to shut those stands down. Lastly, George Henry Thomas is also a very worthy nominee for his demonstration of true patriotism and understanding that victory can only be achieved when one’s opponent admits defeat.  Thomas’ example is actually in keeping with Daniel Pipes’ more recent promotion of the Israel Victory Project.  While victory can only be achieved by the admission of defeat by one’s opponent, Thomas actually failed at achieving that, as Greg concedes.  Some Southerners continue “The Cause” to this day, so it is now our responsibility to complete what Thomas started.

Fortunately, because we are goiter-free and enjoy elevated IQs we are now positioned to pursue the total defeat of The Cause, rocking on YouTube, playing games with other Geeks, understanding human nature, and fighting PLDDers of all sorts.  For this we owe a debt of gratitude to Joy Morton and award him the 2018 Al Copleand Humanitarian Award.

We had so many excellent nominees for The Al this year that I need some extra time to select the winner.

Our nominees include Leo Moracchioli, Richard Garfield, Elizabeth Vandiver, Eric Lundgren, Adam Butler and Autumn Thomasson, George Henry Thomas, and Joy Morton.

As you enjoy your candy you can review all of these nominees and await the announced winner tomorrow.

Education policy leaders have been obsessed with STEM for many years now.  They note the relatively high salaries of students who complete school with STEM skills.  And industry leaders repeatedly complain about the chronic shortages of skilled workers in technical fields.  If only our schools could produce more graduates with these technical skills, we could help address industry’s needs as well as launch students into lucrative careers.

Huge investments have been made to steer students into STEM fields.  Philanthropists have backed coding camps and embraced STEM-focused charters.  And policymakers have poured millions into expanding STEM programs in public schools and universities.  Arkansas has gone as far as requiring that every public and charter high school offer a computer science course so that all students can learn to code.

A fascinating recent paper by David Deming and Kadeem Noray, however, suggests that the payoff to students for pursuing STEM may be short-lived.  STEM workers initially experience elevated salaries and rates of employment, but the skills their occupations require change so rapidly that their training quickly becomes obsolete.  While most workers in other occupations tend to experience a significant rise in earnings as experience enhances their skills, STEM workers tend to have flatter career earning trajectories. As Deming and Noray put it:

We show that the economic payoff to majoring in applied STEM fields such as engineering and computer science is initially very high, but declines by more than 50 percent in the first decade after college. STEM majors have flatter age-earnings profiles than college graduates who major in other subjects, even after controlling for cognitive ability and other important determinants of earnings.

Like professional athletes or movie stars, STEM workers may make a lot of money right out of the gate, but their prospects fade quickly.  If they don’t have non-technical skills to make the transition into management or other occupations, they may suffer the fate of former athletes who couldn’t get an analyst gig or aging actresses who aren’t Meryl Streep.  It’s ironic that the same kinds of education pundits who cluck about how irresponsible it is to offer sports and theater opportunities to students for fear of encouraging them into such high-risk and short-lived careers remain blissfully unaware of the similar (albeit much less severe) career dynamics in many STEM fields.

And as to those severe labor shortages that the tech industry complains about, Deming and Noray say: “Faster technological progress creates a greater sense of shortage, but it is the new STEM skills that are scarce, not the workers themselves.” Tech companies are laying off older workers with slightly older skill sets at the same time that they are starving for new workers with the latest training.  If tech companies want to solve their shortage problem they may need to look in the mirror rather than expect the education system to fix this entirely for them.  They may need to invest more in retraining older workers to keep their skills current.  Or they may need to increase the pay premium for starting workers enough to entice more to take the risks of having a short-lived lucrative career.

While schools still need to do much to improve their efforts in math and science, they should avoid narrowing their focus too much on STEM.  Doing so may serve industry’s insatiable appetite for new, skilled workers, but it may do a long-term dis-service to their students who need a broader set of skills to prosper over their entire working careers (let alone cheating them of the broader education they need to be more enlightened human beings).