Jay P. Greene's Blog

Today AEI released a systematic review by Collin Hitt, Mike McShane, and Pat Wolf on the relationship between changes in test scores and changes in later educational attainment in rigorous studies of school choice programs.  I’ve been writing and talking about this for some time now, inspired to a large degree by informal conversations with the authors of this new report.  Now they have made the point more systematically.

They examined every study of school choice programs with both test score and attainment effects, consisting of “39 unique impact estimates across studies of more than 20 programs.”  They examine whether the direction and significance of the estimated effects of those programs on test scores are consistent with the direction and significance on attainment.  They are not.

They find: “Across the studies we examine, there is no significant or meaningful association between school choice impacts on math scores and high school graduation or college attendance. Nor are ELA impacts meaningfully associated with high school graduation rates. Under some tests, the relationship between ELA impacts and college attendance are significant—but the relationship is weak in magnitude, and the sample of studies is far narrower for college attainment than for high school graduation.”

Keep in mind that the policy relevant question is not whether individual changes in test scores are correlated with individual changes in attainment.  There is some research that has found this relationship (see for example Chetty, et al), but a surprising number of studies find no or only a weak relationship between individual gains on these near-term and later-term measures of success.  But none of them directly address the policy relevant question of whether aggregate test score changes at the school or program level are predictive of aggregate changes in attainment.

If we are going to judge schools or programs as good or bad based on changes in test scores, then those aggregate measures (not individual results) should be predictive of later success.  The fact that they are not, at least when judging school choice programs and schools, suggests that there is something fundamentally wrong with how we have approached public regulation (wrongly called “accountability”) of those programs.  You can’t regulate the quality of schools and programs if you can’t predict their quality.

My student, Ian Kingsbury, will be presenting a paper next week at the annual meeting of the Association for Education Finance and Policy examining factors that help explain which applications to operate charter schools are more likely to be approved.  He is still in the early stages of this project and I’m sure will benefit from feedback on how to improve the work, but he has already analyzed nearly 400 applications to operate charter schools in 7 states.  His basic findings, which seem unlikely to change as he gets feedback, should surprise no one but should shock everyone interested in charter schools — the more burdensome the regulatory environment for approving charters, the less likely charters led by minority applicants are to be approved by authorizers.

Using the score that the National Association of Charter School Authorizers (NACSA) gives to each state’s charter policies as a proxy for regulation, Ian finds that for each 1 point increase in a state’s NACSA score (on a scale from 0 to 33), African-American and Hispanic-led charter applications are 1.7 percentage points less likely to be approved.  Given that one state included in the study has had a NACSA score as low as 9 and other states, like Indiana and Nevada, have a 33, the variation in regulatory environments Ian observed is associated with about a 41 percentage point difference in the probability that minority-led charter applications would be approved.

Of course, the charter regulations favored by NACSA, the National Alliance of Public Charter Schools, and most of the charter establishment are meant to promote quality.  Unfortunately, there is no evidence that these regulations are in fact associated with higher quality charter schools. But they clearly make it harder for charter applications to get approved, especially for minority-led charter applicants. Even when Ian controls for whether the minority-led charter applicants attended more selective colleges or were affiliated with CMOs/EMOs, which might be proxies for the quality of those applications, minority-led applicants were still 1.7 percentage points less likely to get approved for each 1 point increase in NACSA score.  In other words, if the purpose of regulations is quality control, those regulations still tend to keep out minority-led charter schools even after adjusting for reasonable proxies of the quality of those proposed charter schools.

This pattern of regulations in the name of quality posing a disproportionate barrier to minorities without actually being related to quality should sound familiar to anyone who has paid attention to the issue of occupational licensure.  A variety of groups, from the Obama White House to the Institute for Justice, have noted that raising requirements to enter many occupations has been an important barrier to opportunity, especially for disadvantaged groups.  Requiring people to spend 2,100 hours and about $22,000 to obtain a cosmetology license before they can braid hair has little to do with quality but is an important obstacle to opportunity.  The same can be said of the type of regulations favored by NACSA and the charter establishment — they have little to do with quality but seem to be large obstacles to minority operated charter schools.

Keeping out minority-led charter schools has potentially serious educational and political implications.  There is some evidence that minority students fare better when educators are of their same race/ethnicity.  Minority-led charter schools may be more likely to provide this type of educational benefit for minority students.  In addition, excluding minority leaders of charter schools severely damages the political prospects for charter schools by making minority community leaders significantly less invested in the growth and success of the charter sector.

If any of you will be at the Association for Education Finance and Policy conference next week, I would encourage you to stop by Ian’s panel on Thursday (March 15) at 10:15.  While Ian’s project is not finished, the evidence is becoming clear enough that NACSA and the rest of the charter establishment need to explain why the policies they favor have such a negative effect on minority-led charter schools.  And if they are going to defend that negative effect by claiming that the policies they favor promote quality, they need to provide evidence to support that claim.  The way it looks now, the types of regulations favored by NACSA and others seem to just keep minorities out without producing any increase in quality.

Update — I’ve added a link to the paper, which is available here.

(Guest Post by Gema Zamarro & Lina M. Anaya)

Employment in the so-called STEM (Science, Technology, Engineering and Mathematics) fields is projected to continue growing according to the Bureau of Labor Statistics. Additionally, wages in STEM field occupations are estimated to be on average nearly double the national average of wages for non-STEM jobs. Despite this promising future, women continue to be under-represented in STEM. Women are less likely to enroll in STEM degrees in college and represent a smaller share of STEM occupations. The question is why? Only after understanding the possible sources of such gender gaps we can have an idea of what can be done about it.

This question has haunted me (Gema) since my daughter, then a kindergartener, came home one day saying a boy in her class told her “girls are not good in math.”  Indeed, researchers have pointed out at gender differences in math performance and math perceived ability as possible drivers of later gender gaps in STEM. I wondered if parents could somehow counter these effects. After all, my previous work indicated that parental occupation type could be important for women’s long term STEM outcomes. In a recent working paper, I partnered with Lina M. Anaya, a Ph.D. student at the Department of Education Reform at the University of Arkansas, and Frank Stafford, Economics professor at the University of Michigan, to try and shed some light on these questions, using data from the Panel Study of Income Dynamics (PSID).

Using information from the multiple supplements of the PSID, we measured gender gaps in performance  on the standardized Woodcock  Johnson Applied problems test (W-J AP) and self-reported perceived math ability, measured on children from PSID families when they were between 6 and 17 years old (on average around 11 years old). Then, we are able to track these children and study their likelihood of majoring in a STEM field in college. We found that boys are more confident in their abilities than is warranted by their performance, while girls are less confident than is warranted by their performance.  But the problem isn’t just one of lack of confidence — boys’ confidence contributes more to their pursuit of STEM majors than girls’ confidence, even if they had the same true ability and same level of confidence.

Our results corroborated significant gender differences in W-J AP test performance and in perceived math ability during childhood. Even after conditioning in a given level of math performance in the W-J AP test, girls reported significantly lower levels of perceived math ability than boys (See Table 1). In the highest percentiles of math performance, 64% of boys reported the highest levels of perceived math ability, as compared to 50% of the girls. Even in the lowest levels of math performance, boys tended to be more optimistic with respect to their math ability, 29% of the boys reported the highest levels of perceived math ability, relative to 17% of the girls doing so. Having a parent with an occupation in STEM helped increase math performance but did not seem to help improve perceived math ability, if anything it seemed that those with parents in science were more pessimistic.

Table 1: Perceived Math Ability by Gender, given W-J AP scores (% of sample)

Note: Weighted percentages reported using child population weights

Interestingly, girls’ lack of perceived ability seems to be something specific to math and not the result of girls generally reporting lower levels of perceived ability. The PSID also included results in the Woodcock Johnson reading test (W-J reading) and asked kids to report on their perceived ability in reading. We use this information to study perceived ability in reading conditional on performance. As it can be seen in the results in Table 2, gender patterns are very different for reading, a subject where girls, on average, outperform boys. In this case, we observe smaller gender differences of perceived reading ability among those scoring in the higher percentiles of the W-J reading test while girls performing in the lower percentiles report higher levels of perceived ability than boys.

Table 2: Perceived Reading Ability by Gender, given W-J reading scores (% of sample)

Note: Weighted percentages reported using child population weights

Finally, since the PSID tracked these kids, we study to what extent math performance and perceived math ability, during childhood, and parental occupation type are related to the probability of majoring in STEM during college. Overall, as expected, we find that women are less likely to major in STEM in our sample, especially when we look at the so called “hard sciences” fields of engineering, architecture, mathematics and computer sciences. Both higher levels of math performance in the W-J AP test and higher levels of perceived math ability are related to higher probabilities of majoring in a STEM field.

But, here is where it gets interesting, the effects of higher levels of math performance and perceived ability are much bigger for boys than for girls. Performing in the highest percentiles of the W-J AP distribution, as compared to performing in the lowest percentiles, is associated with an increase in the probability of majoring in a “hard sciences” STEM field of about 13 percentage points for boys but only 6 percentage points for girls. Similarly, reporting the highest levels of perceived math ability, as compared to the lowest levels, is associated with an increase in the probability of majoring in a “hard sciences” field of about 7 percentage points for boys but only 2 percentage points for girls. These results suggest a loss of STEM enrollment by otherwise capable women.  And we can’t simply fix the problem by trying to boost women’s confidence in their true abilities, because women’s confidence contributes less to pursuing STEM than men’s confidence.  Perhaps men are rewarded for over-confidence in a way that women are not.

Interestingly, having a parent who works in a STEM occupation could help girls and not so much boys. The probability of majoring in “hard sciences” STEM fields increases by about 14 percentage points for girls when one of the parents works in a science job. For boys the increase of this probability is only 4 percentage points. Whatever the reason, these results suggest that parental occupation type could be an important factor reducing gender differences.

As for the answer I gave to my daughter, I said “It is not true that girls are bad at math. Look at your mother. My job is doing math all day!” I work on the field of applied econometrics and so, I guess that was close enough.

In a fascinating new study by Robert W. Fairlie and Peter Riley Bahr, they examine the effects of an experiment in which some community college students received free computers and others did not by lottery.  Comparing these randomly assigned treatment and control groups, the researchers found that computer skills rose among students who were given computers, but those skills did not translate into higher college enrollment, employment, or earnings for the treatment group.

These results are particularly important because many politicians have focused on improving computer skills as the key to improving educational outcomes.  In Arkansas, the main education policy initiative championed by the governor is a law that requires all public schools to offer computer science classes.  Texas has adopted a similar policy.  Leaving aside all of the obvious practical concerns, like whether schools have or can develop staff qualified to teach computer science, this new research raises questions about the aim of these policies.  How important is increasing computer skills for the vast majority of students?  No one doubts that most workers have to use computers, but many students may already possess the skills they need and it seems doubtful that raising average computer skills would lead to significant changes in employment outcomes — and that’s assuming we can improve computer skills in a meaningful way.

The new study is also incredibly useful in that it reminds us of how important it is to rely on randomized experiments rather than studies that use matching or controls for observables.  They conclude:

Importantly, our null effect estimates from the random experiment differ substantially from those found from an analysis of CPS data, raising concerns about the potential for selection bias in non-experimental estimates of returns.  Estimates from regressions with detailed controls, nearest-neighbor models, and propensity score models all indicate large, positive, and statistically significant relationships between computer ownership and earnings and employment, in sharp contrast to the null effects of our experiment.  It may be that non-experimental estimates overstate the labor market returns to computer skills.

It is simply false that matching studies are just as good or almost as good as randomized experiments.  Sometimes you get the same result in a matching and RCT study, but that could simply be because selection did not bias the result in that case or you were just lucky.  Sometimes a coin flip will also give you the same result.  Theoretically, we know that selection bias is a serious concern, which means that we can never have strong confidence in research designs that assume selection issues don’t exist.

(edited slightly)

Matt Ladner and Max Eden have observed that the National Alliance for Public Charter Schools (NAPCS) as well as the National Association of Charter School Authorizers (NACSA) give their highest ratings to states that barely have any charter schools.  As Max noted of the NAPCS rating: “Half the states in the top ten have fewer than 10 charter schools.” He added: “three of your top 10 states have produced 20 schools in 20 years.”

It’s bad enough that there is no evidence to support the criteria that NAPCS and NACSA use to rate state charter laws, but it’s intolerable that their ratings seem to praise policies that are associated with slowing the spread of charter schools — the exact opposite of what these groups are supposed to be advocating for.

It’s nice that Robin Lake is noticing that charter growth has stalled and that Portfolio Management is beginning to block more charter schools in Denver, but somehow neither she nor Paul Hill, nor any of the other charter intelligentsia, seem to be able to connect the dots and trace the problem to the types of burdensome regulatory schemes that they all prefer.  These folks back burdensome regulations with the claim that they help promote charter quality and forestall political problems, even though there is no evidence that they accomplish either of these goals.  But the evidence is becoming quite clear that what these burdensome regulatory schemes accomplish is the creation of very few charter schools and even fewer led by minority members of local communities.

It’s as if the charter intelligentsia thinks that the best charter school is the one that isn’t there.  This reminds me of the Girlfriend in Canada trope.  The best girlfriend (charter school) is the one who isn’t there.  She’s really great and I wish you could meet her, but she lives far away. Avenue Q captured this trope nicely, so I’ve modified the lyrics a bit:

I wish you could go to my charter school
My charter school that’s placed in Canada
The scores couldn’t be higher, I swear I’m not a liar
My charter school that’s placed in Canada
Its leadership is Ivy League, too bad they’re all lily-white
Competitors are not in sight, no one can put up a fight
They test kids every single day, just to make sure that everything’s okay
It’s a pity the school’s so far away in Canada
Last year we reported the highest grad rate
Too bad it’s because we chose to inflate
It’s so sad, that doesn’t mean we’re not great
Our discipline’s progressive and our politics transgressive
I wish you could go to my charter school
But you can’t because it’s in Canada
I know I’m persistent, even if it’s non-existent
That’s why I favor district schools… er, I mean charter schools
Darn, I really want district schools to create more charter schools
It’s the best charter school, my wonderful charter school
Yes, I have a charter school that’s placed in Canada
And I can’t wait to give kids more choices

It’s finally out!  Run, don’t walk, to the Amazon or the Rowman and Littlefield web sites to buy your copy.

The book was a fascinating exercise that produced some really useful results.  Mike McShane and I asked 9 very smart and experienced education scholars each to identify a failure in education reform and reflect on why that failure occurred and what could be learned from it.  We didn’t pickwhat policies were failures, we let the authors do that.  The only thing we did was encourage people to engage in some useful self-criticism and try to focus on policies with which they were sympathetic.

Be sure to check out the chapters by Rick Hess and Paige Willey, Larry Cuban, Dan Willingham, Marty West, Ashley Jochim, Matthew DiCarlo, Anna Egalite, Matt Ladner, and Megan Tompkins-Stange.  And in the intro and concluding chapters Mike McShane and I try to identify common themes and lessons across these contributions.

Rather than blaming others, this book is about the honest mistakes we all make in trying to improve education policy and how we might avoid similar mistakes in the future.  There is nothing inherently wrong with failure.  The problem is our unwillingness to acknowledge and learn from failure.  If we can’t do that we tend to repeat the same policy failures over and over.  I would rather that we “fail better.”

Or as Samuel Beckett put it:

“All of old. Nothing else ever. Ever tried. Ever failed. No matter. Try again. Fail again. Fail better.”

Paul Peterson has an excellent interview of Mo Fiorina on Education Next regarding Mo’s new book.  This new book, like several earlier works by Fiorina, makes the case that America is not coming apart at the seams, despite appearances.  Based on a careful analysis of public opinion polls and his extensive knowledge of American politics, Fiorina argues that Americans are no more divided on political issues now than they have been for many decades.  Most Americans remain moderate in their politics and rarely embrace extremist views or movements.

The reason things feel more divided is that political parties have become more homogenous internally and more distinct from each other.  Gone are the Southern, conservative Democrats, like George Wallace or even Sam Nunn, and the liberal, Northeast Republicans, like Nelson Rockefeller and Edward Brooke.  The Voting Rights Act ended one-party rule in the South and other regional issues have given way to parties with uniform, national agendas.  Parties have also become more responsive to national donors, who fund campaigns and drive the agenda in local races throughout the country.  This has made the country more partisan, but not more divided, since people have just been sorted more clearly into distinct parties.

Paul asks Mo an excellent question: how come one of the parties hasn’t moved its positions closer to the middle to capture all of those moderate voters and ensure greater electoral success?  Mo answers that parties are no longer primarily about winning elections.  They are primarily concerned with articulating and promoting the more extreme views of their donor and activist bases.  Paul and Mo were my graduate advisors and I served as a teaching and research assistant for both, so I am always inclined to believe them.  Despite my prejudices, however, I think Mo makes a persuasive case that has implications for the ed reform movement.

What if ed reform foundations and organizations are not, for the most part, really concerned with winning?  What if, like political parties, they are just trying to articulate and promote the worldviews of their donors and activist bases?  Thinking about ed reform foundations and organizations like Fiorina thinks about political parties would explain a lot.  It could explain why foundations and the organizations they fund have pursued a series of reforms whose failures were easily predictable, from Measuring Effective Teachers to Common Core to Portfolio Management to reforms that focus narrowly on the most disadvantaged.  It could also help explain why people at foundations and reform organizations almost never experience consequences when the ideas they back fail.  Ed reformers are so far removed from accountability that Tom Vander Ark was even pushed out of Gates for backing a strategy that succeeded.

Maybe many ed reform foundations and organizations are not actually about reforming education as much as they are about appearing earnest in support of things elites consider to be good.  This would help explain the disproportionate amount of energy devoted to posturing on social media and making speeches to each other at conferences.

Of course, it’s easy to become too cynical about ed reform, just as it is too easy to despair about the nation’s political divides.  But Mo offers a parsimonious theory that not only explains why parties have not moved more to the middle but also why ed reform appears stuck with a string of political failures.

I am pleased to announce that the Department of Education Reform at the University of Arkansas will have two new faculty members.  Both Jonathan Wai and Albert Cheng will be joining us in August 2018 as Assistant Professors in Education Policy.  Jonathan Wai will also hold the title of Endowed Chair in Education Policy.  The current holder of that endowed chair, Gary Ritter, will be leaving in August to become the Dean of the School of Education at St. Louis University.

Jonathan received his B.A. in Psychology and Mathematics from Claremont McKenna College followed by an M.A. in Psychology and Evaluation from Claremont Graduate University.  He then earned his Ph.D. in Psychology from Vanderbilt University.  Following that, Jonathan was a post-doc at Duke University and remained there as a Research Scientist affiliated with Duke’s Talent Identification Program (TIP), which strives to identify and offer enriching opportunities to gifted students, especially those from disadvantaged backgrounds.  Jonathan’s research covers a variety of topics, such as gifted programs and the role of intelligence in educational success, and has been widely cited.

Albert received his B.A. in Mathematics from the University of California at Berkeley followed by an M.A. in Education from Biola University.  He taught math in a school in California before enrolling in and completing his Ph.D. in Education Policy from the University of Arkansas.  Following that, Albert was a post-doc at Harvard University in the Program on Education Policy and Governance.  Albert’s research covers a variety of topics, including path-breaking ways of measuring non-cognitive or character skills and the role of mission in school success.

We are sad to bid farewell to Gary Ritter but extremely excited about these new additions to our faculty.

Jerry Muller has a piece in the Wall Street Journal that should be required reading for foundation staff and ed reformers who are obsessed with metrics.  Here is a snippet:

Metric fixation consists of a set of interconnected beliefs. The first is that it is possible and desirable to replace judgment with numerical indicators of comparative performance based on standardized data. The second is that making such metrics public (transparency) assures that institutions are actually carrying out their purposes (accountability). Finally, there is the belief that people are best motivated by attaching rewards and penalties to their measured performance, rewards that are either monetary (pay for performance) or reputational (rankings).

But not everything that is important is measurable, and much that is measurable is unimportant. Most organizations have multiple purposes, and that which is measured and rewarded tends to become the focus of attention, at the expense of other essential goals. Similarly, many jobs have multiple facets, and measuring only a few of them creates incentives to neglect the rest. Almost inevitably, people become adept at manipulating performance indicators. They fudge the data. They deal only with cases that will improve performance indicators. In extreme cases, they fabricate the evidence.

It’s not that measurement is useless or intrinsically pernicious. The challenge is to specify when performance metrics are genuinely useful—that is, how to have metrics without the malady of metric fixation….

Tools of measurement are most useful for internal analysis by practitioners rather than for external evaluation by the public, which may fail to understand their limits. Such measurement can be used to inform practitioners of their performance relative to their peers, offering recognition to those who have excelled and offering assistance to those who have fallen behind. To the extent that they are used to determine continuing employment and pay, they will be subject to gaming the statistics or outright fraud….

Just because performance measures often have some negative outcomes doesn’t mean that they should be abandoned. They may still be worth using, despite their anticipatable problems. It’s a matter of trade-offs, and that too is a matter of judgment.

With measurement as with everything else, recognizing limits is often the beginning of wisdom. Not all problems are soluble, and even fewer are soluble by metrics. It’s not true, as too many people now believe, that everything can be improved by measurement, or that everything that can be measured can be improved.

In education reform, like other policy areas, analysts are busy trying to identify how to channel people’s behavior in directions that we believe will improve their lives.  If we think people should eat less, we devise interventions to encourage them to cut back.  If we think too few students go to college, we nudge them to enroll.  If we think people save too little, we arrange systems to increase retirement contributions.  In what is imagined to be a kinder, gentler approach to these problems, we “nudge” people toward desired outcomes rather than mandating them.  Mandating can seem too harsh and produce backlash, but nudges allow social scientists to influence behavior without feeling like they are infringing on the autonomy and liberty of the people whose behavior they are shaping.  Most people behave irrationally and lack impulse control, but the priestly class of social scientists can detect and correct these problems for other people.

A central problem with this approach is that we know too little about the lives of others to know with confidence what is good for them or how our nudges will affect their entire lives.  We may think we are helping people, but absent the same contextual information that individuals possess about themselves we are liable to push (excuse me, nudge) people in ways that actually harm them.

A recent NBER study on a plan to increase retirement savings helps illustrate the challenges associated with managing other people’s lives.  That study examined a natural experiment in which civilian employees of the military were automatically enrolled in contributing 3% of their income toward a retirement plan that would be matched by employer contributions.  Previously, employees had to opt-into this retirement plan, but the policy changed so that employees would have to opt-out if they did not wish to contribute.  Already employed workers were exempted from the change, so researchers could compare those hired just before the policy was implemented to those hired just after to identify the causal effect of the switch toward automatic contributions.

As intended, the plan increases retirement savings: “At 43-48 months of tenure, automatic enrollment increases cumulative employer plus employee contributions since hire by 5.8% of first-year annualized salary.”  While the plan succeeded in increasing retirement savings, employees did not appear to decrease their consumption.  Instead, they increased their borrowing, particularly for cars and homes, by an amount that exceeded the amount by which their retirement savings increased.  Even if we exclude the mortgage borrowing, which has a more ambiguous affect on long-term wealth given that house prices may appreciate by more than interest and depreciation, even just the auto loan increases exceeded the amount by which employees increased their savings.  If we include the employer match, increased retirement wealth was close to the increase in auto loan amount.  Overall, it is unclear if the policy increases the total wealth of the employees.  Excluding the employer match, employee wealth is likely decreased. So the only clear effect of the automatic savings policy is a transfer of wealth from whoever pays for the employer match to employees, but no overall benefit to society.

As this study reveals, shaping other people’s behavior is complicated.  People may not contribute to retirement plans because they want to consume things now.  Pushing (er, I mean, nudging) them to save anyway may just cause them to increase their borrowing in a way that has no net benefit or even a net harm.  We don’t know enough about other people’s lives to manage them optimally.

But retirement savings is usually considered one of the clear success stories for nudges.  I suspect that is because people typically judge those interventions by the extent to which they change the narrow behavior on which we are focused rather than overall well-being. That is, we nudge people to contribute more to retirement savings and sure enough they do.  Mission accomplished.  But we really need to look at their total consumption and borrowing behavior to see how this policy affects folks.  When we step back to see the bigger picture, the benefits disappear or even become harms.

The same is true for educational nudges.  We have a number of studies that look at short-term and narrow effects of nudges to get students into college.  Sure enough, if we push (I mean, nudge) people to enroll in college, they tend to do that.  All that shows is that people believe we are experts and are willing to substitute our expert advice for them (even though we know almost nothing about them) for their own, better informed judgement about what they should do.  The real proof of college-going nudges is not whether people listen to us, but whether that helps them long-term.  Those long-term results have not yet been published, but those results exist and I believe based on leaked drafts that the short-term benefits go away or even turn into harms after a few more years.   That is, students who didn’t think they were ready for college but were pushed into attending may have difficulty finishing and other students who enroll later may be better prepared at that point to succeed, causing the overall effect of these nudges to be null or even negative.  One has to wonder about the ethics of researchers touting short -term positive results if they know that longer-term effects tend to go away or turn negative.  Are they nudging us to accept the idea of nudge interventions, so it is really for our own good to only hear about positive short-term results?

(H/T Bob Costrell for alerting me to the retirement study.)