Chutzpah is jokingly defined as murdering one’s parents and then complaining about being an orphan. Atila Abdulkadiroglu, Parag Pathak, and Christopher Walters ( hereafter APW or the MIT team) sure show some chutzpah when complaining about not having continued access to data regarding the Louisiana Scholarshp Program (LSP) in a recent article. While I don’t know for sure why they were denied continued access to data, I believe that it is related to their rush to release 1st year results from their evaluation. Why they were rushing is an incredibly depressing story about how status and power in our field contributes to academic abuse and dishonesty– a story the reporter who wrote the article entirely missed.
It is not widely known or acknowledged, but the original analysis of 1st year result from LSP was conducted by Jonathan Mills when he was a doctoral student along with his advisor, Patrick Wolf, at the University of Arkansas. They presented those findings at academic conferences 8 times during 2014 and 2015 and they were contained in Jon’s dissertation published in July 2015. APW were at some of those conferences. Atila actually had lunch at one conference with Jon and Pat during which they discussed that study in June of 2015. Atila never indicated that he was conducting or planning to conduct a similar study. He offered to help and they sent him some materials. He never responded with help but he did move forward with his own study with the MIT team without informing Pat or Jon that they were doing so.
APW released their own study as an NBER report in December 2015. Nowhere in that report did they acknowledge or cite Jon and Pat’s earlier work of which they were almost certainly aware, having discussed it with them. Nor did APW acknowledge that their study was essentially a replication of Jon and Pat’s earlier study. The research designs were nearly identical. The data were almost the same. The only difference was that Jon and Pat had a more complete data set and as a result reported more negative results.
That’s right. Jon and Pat had more negative results. They released those results along with the negative 2nd year results in February 2016. So the fact that Jon and Pat continued getting access to LA data while APW did not does not appear to have anything to do with reporting negative results. It seems to be related to the fact that APW were rushing to release results. They didn’t take the time like Jon and Pat did to solve missing data issues. Instead they were determined to move fast to get their results out first.
Why did it matter that they be first? By being first to release they could act like they had the original analysis rather than a replication. Top Econ journals tend not to be as interested in replications of a grad student’s dissertation. And by being first to release and not citing Jon’s work they could act like theirs was the original analysis.
Failing to credit and cite earlier work is a form of academic fraud. I have not come forward earlier with this story because Jon was entering the academic job market and did not want to get on the wrong side of high status and powerful people in the field. Pat and I, as his advisers, deferred to his wishes and remained quiet. Now that Jon has a secure job ( with us) and a news article wrongly implies that APW were denied access because (presumably unlike us) they wouldn’t withhold negative result, I felt compelled to tell this story. It’s an ugly one.
UPDATE: Pat Wolf checked his records and found that he also had a discussion at a conference in April 2015 with Atila regarding the Louisiana evaluation that he and Jon were doing. The materials he sent, however, were following that conversation, not following the June conversation as Pat had earlier remembered, and those materials were not directly related to the study. In any event, it is clear from multiple conversations and multiple conference presentations that APW were aware of the existence of prior research.
2nd Update: APW have a statement here. My response to it is here.
(Edited for typos and to add links)
Jay P. Greene's Blog 
Excerpt from the 74 article:
“In October 2015, Pathak emailed two DOE staff members the results from his soon-to-be-published study with a pair of colleagues showing that students who received a state-funded voucher to attend a private school saw large drops in test scores after one year compared with students who had applied for a voucher but didn’t get one. Hours later, Brian Darrow shared the results with John White and asked if he wanted to join a debriefing call, according to department emails.
An email three days later to Pathak from DOE staffer Gabriela Fighetti shows the conference call happened and included White.
“I know it wasn’t the easiest of calls,” Fighetti wrote, not indicating to what difficulty she was referring.
Fighetti also suggested that if the researchers would hold the paper, the department might be able to provide them additional data based on more years. Pathak welcomed the opportunity.
“We would be happy to include additional years of data,” he wrote in response. “We all think the study would benefit from this and would be willing to wait to release this study provided we could obtain the outcomes in a relatively short time frame.”
Fighetti soon emailed Darrow saying they should discuss with White whether to let the researchers publish with a single year of data or to provide them more years. They decided on the latter.”
****
Gee, I wonder why the MIT/Duke/Berkley team was in such a rush!
Another excerpt below. Clear and irrefutable evidence that the MIT/Duke/Berkeley team losing their data-share agreement had NOTHING to do with their results. Also, credit where credit is due: Harris is 100% right here (and his use of the word “unfortunate” is clearly an understatement).
****
Doug Harris, the director of the Research Alliance and a Tulane University professor, wrote to White in a Jan. 10, 2016, email, “We should talk at some point about the voucher studies and what [the MIT and Duke researchers] did. The way the authors handled this is unfortunate. You can trust that we would never do anything like that.”
Asked to elaborate, Harris told The 74: “I thought it was unfortunate that the earlier study was released with only one year of data, when there was good reason to think the results would be different in the second year. My understanding of the situation from LDOE was that the researchers were granted access to [a] second year of data, but that they wanted to release the results quickly.”
“In the email, I was communicating to Mr. White that this is not how we operate at the research center I direct, that we wouldn’t release results quickly if there was good reason to think, with additional feasible analysis, that the conclusion might change.”
Harris says that he feels no obligation to frame studies in a certain way to get continued access to data.
“We feel no pressure at all,” he said. “I think that’s fairly obvious since we’ve released reports that directly conflict with LDOE’s and [the state Board of Education’s] most closely held positions, especially on vouchers.”
It seems reasonable that NBER (run out of MIT) should release details of the full timeline of the submission. If authors are being sincere, it should show the paper was submitted in late December.
Wow! Thank you for having the courage to share this story. Exposing this kind of deeply unethical behavior is a real public service.
Thanks for the story.
[…] How Three MIT Economists Stole Idea from a UARK PhD student and Published It as Their Own https://jaypgreene.com/2017/08/04/the-chutzpah-of-abdulkadiroglu-pathak-and-walters/ […]
there *is* a citation to Mills and Wolf in the NBER paper in a footnote, although bizarrely the paper doesn’t appear in the references.
Dear Professor Greene,
This post contains some serious allegations about our work on the Louisiana Scholarship Program. We have written a response to these allegations, available here:
Click to access greene_memo.pdf
Sincerely,
Chris Walters
My concern is the failure to cite and give credit to previous work. I notice that the APW reply does not deny the existence of prior work, both conference papers and dissertation chapter. It also does not deny that they were aware of that earlier work. Nor does it deny that they failed to cite that work in their NBER report. In short, the reply says many things but does not address the central concern I have raised.
Your team’s research is not “prior work.” As shown by the date on the data agreement provided in our response, the two projects were in progress simultaneously.
We would have cited a public working paper had we known of one in December 2015. We revised our paper to cite Mills/Wolf (2016) within a month of their paper’s release, as can be seen here:
Click to access w21839.rev1.pdf
We were not aware of the Mills dissertation chapter and, as our response notes, your team’s work also did not cite this chapter until 2017.
“Fraud” is an irresponsible word to use for a situation in which two research teams are simultaneously engaged in research on the same topic. We request that you retract your accusation of academic fraud.
We will not be responding to any more statements on this blog. Feel free to reach out by phone or email if you would like to discuss the situation further.
Prof. Walters,
So you’re saying that Pat Wolf is lying when he says that he told you about his study in April and June 2015? And that he has forged written records to that effect?
“We will not be responding to any more statements on this blog. Feel free to reach out by phone or email if you would like to discuss the situation further.” translates to “publicly answering the specific questions you raise would not reflect well on us.”
Emmett, that translation seems accurate. See here for Jay’s further commentary on this unfortunate matter: