In my last post I described how Atila Abdulkadiroglu, Parag Pathak, and Christopher Walters (APW) released an evaluation of the 1st year results from Louisiana’s voucher program through NBER dated December 2015 (although actually released in January 2016) that failed to cite or provide appropriate credit to earlier conference presentations by Jonathan Mills and Patrick Wolf and a dissertation by Mills. In response APW issued a statement that raises many issues, but fails to address the heart of the matter. Before replying to some of those extraneous issues, let’s focus on the key questions:
- Did Jon and Pat conduct analyses, write papers, and present findings to the academic community of the same program using the same basic methodology and data as APW prior to their December 2015 NBER paper?
- Were APW aware of this prior work?
- Did APW fail to cite and give appropriate credit to that prior work in their December 2015 NBER paper?
If you read their statement closely you will see that APW do not deny the existence of prior work, do not deny being aware of that work, and do not deny failing to cite it. Let’s take a look at what they did say.
The No Prior Working Papers Claim
APW do not deny that Jon and Pat had multiple conference papers, starting with one to the Association for Public Policy Analysis and Management on November 8, 2013 followed by 8 more during 2014 and 2015. Instead they focus on whether there were prior working papers: “Mills and Wolf released a working paper in February 2016. This is the first working paper by the Arkansas team that we are aware of.” Christopher Walters reiterates this point in a comment on the blog: “We would have cited a public working paper had we known of one in December 2015.”
There are two problems with this no prior working paper argument. First, a working paper is not the only format of prior academic work that requires citation. There is no exemption to the scholarly obligation to credit prior work if that work is in the form of conference papers and presentations. In fact, the American Economics Association helpfully provides guidance on the appropriate way to cite “Lectures and Papers Presented at Meetings.”
Second, there is a very good reason why Jon and Pat did not have a working paper for APW to cite and APW are almost certainly aware of that reason: The Louisiana Department of Education (LDE) asked Jon and Pat not to widely disseminate their findings until after the second year of results were complete. LDE had made the same request of APW, who similarly complied with that request before deciding that it would take them too long to finish the 2nd year analyses. As APW describe it in their statement: “We postponed the release of our paper because the LDE promised us additional data in exchange for delaying public disclosure of our results. We released the paper when we judged that this data was not forthcoming.”
While Jon and Pat did not widely publicize the results of their 1st year evaluation by issuing a working paper online, they did not keep their findings a secret. They presented their results at numerous conferences and in a dissertation, and they solicited feedback and suggestions from colleagues (including from Abdulkadiroglu). And they took that time to resolve missing data issues, complete the 2nd year analyses, and release both 1st and 2nd year results in a working paper only a month after APW released their 1st year results.
By contrast, APW operated in a different manner. As far as I can tell they did not present their work at any meeting of a professional association prior to December 2015. They did not disclose to Pat and Jon that they were planning to conduct or were already conducting their own study, even when Pat and Jon discussed the research with Abdulkadiroglu. They also did not take the time to work out data issues with LDE and wait for the 2nd year of results before releasing their NBER report. And the failure of that report to cite and credit the work produced by Jon and Pat cannot be excused because Jon and Pat didn’t put a working paper online.
The Data Sharing Agreement Claim
APW’s statement does not deny being aware of prior work by Jon and Pat. In Walter’s comment on the blog, however, he does say “We were not aware of the Mills dissertation chapter …” But Abdulkadiroglu, not Walters, was at the lunch with Jon and Pat in June 2015. The statement they issued together does not deny that Abdulkadiroglu was told about Jon’s dissertation work at that lunch. Instead it denies that Jon’s dissertation was the origin of APW’s work: “This conversation centered on the use of school assignment mechanisms for program evaluation. The conversation occurred more than two years after we signed our data agreement to evaluate the LSP using lotteries, so it was clearly not the inspiration for our work.”
Whatever inspired their work does not obviate their scholarly obligation to cite and give appropriate credit to academic work on the same program that had been produced prior to their December 2015 NBER report. The existence and date of APW’s data sharing agreement with LDE is irrelevant to whether they failed to cite and give appropriate credit to previously produced work.
But in a comment on the blog, Walters offers a novel interpretation of what “prior work” means: “Your team’s research is not ‘prior work.’ As shown by the date on the data agreement provided in our response, the two projects were in progress simultaneously.” By prior work I mean research findings that had been produced and shared with the academic community before December 2015. The fact that both research teams had data sharing agreements does not erase the fact that Jon and Pat had produced, presented, and published (in a dissertation) results prior to the December 2015 NBER report.
In addition, it should be noted that Walters’ description that “the two projects were in progress simultaneously” is very different from APW’s characterization of Jon and Pat’s work as a “followup analysis” in the footnote they added to their NBER report after Jon and Pat released their February 2016 working paper with the 1st and 2nd year results. It should be further noted that even that amended paper does not provide a proper citation because Jon and Pat’s work is missing from the reference section.
Even though the existence and date of the data sharing agreements is irrelevant to the scholarly obligation to cite prior work, the suggestion that APW had an agreement that pre-dated Jon and Pat’s by two years is incorrect. Jon and Pat started negotiating a data sharing agreement with LDE during the fall of 2012, around the same time as APW, and concluded that agreement on January 8, 2013, two months after APW. Jon and Pat were completely unaware of the existence of APW’s data sharing agreement or that APW were pursuing a similar line of research with the same data.
The data sharing agreements are further irrelevant because they do not establish when the research teams actually started the research in earnest. We know that Jon and Pat had started their research during 2013 because they presented preliminary findings at the APPAM conference in November 2013. The earliest we know APW had written-up results, according to reporting by The 74 Million, was in October 2015, when they presented them to LDE. And when Jon and Pat discussed their research with Abdulkadiroglu in June 2015, it is unclear whether he failed to disclose that they were working on a similar study because APW had not actually started that work yet.
The You Didn’t Cite the Dissertation Either Claim
Finally, the APW statement seems to suggest that they are somehow absolved of the responsibility of citing Jonathan Mills’ dissertation because Jon and Pat also failed to cite that dissertation in some of their subsequent papers, including the February 2016 working paper with 1st and 2nd year results. First, it is an entirely different thing to deny oneself credit than to deny someone else credit. Whether Jon and Pat made an error in failing to credit all of their own prior work does not excuse APW in making that error about someone else’s work. This is especially the case because Jon and Pat’s failure to cite Jon’s dissertation does not mislead the reader about who had been the first to produce these results. Second, Jon and Pat’s February 2016 working paper did cite a series of their own prior conference papers from 2014 and 2015, so they did clearly establish that they had been presenting results on the Louisiana program well before December 2015. APW might have noticed those references and acknowledged that record of prior work when they added the footnote in an update to their December 2015 NBER paper mentioning Jon and Pat’s February 2016 working paper.
Raising this issue is certainly not pleasant. In fact, it’s down-right nerve-wracking and I can completely understand why Jon was reluctant to press this matter earlier. But I think credit is being given to other researchers for being the first to produce an evaluation of achievement effects from the Louisiana voucher program that properly belongs to Jon and Pat. I actually tried to resolve this amicably with Parag Pathak when he came to give a lecture at the University of Arkansas in 2016. Parag had been invited prior to the December 2015 NBER report and his visit was uncomfortable. It would have been more uncomfortable if either Jon or Pat were there, but Jon had already left for a post-doc at Tulane and Pat had another obligation. Contrary to the APW statement, I did raise the bruised feelings over credit and data access with Parag and suggested that he might smooth things over with Jon and Pat and perhaps regain access to LDE data if he were to suggest collaboration on some future project with them. I suggested at the very least he should call Pat and talk to him. He never did.
Jay P. Greene's Blog 
The so-carefully parsed words of APW are an object lesson in the craft of deflection and misdirection. Statements that have some narrow technical “truth” but in fact evade the truth. In other words, classic half truths.
It’s unclear what’s at issue here, outside of egos of academic researchers. Two teams had data-use agreements with Louisiana and produced two studies. The studies do not read the same. The study that was first released had negative findings after one year.
The study released a few months later had negative findings after one *and* two years.
None of the team members altered the data or put their names on papers they did not write. Claims of ‘academic fraud’ are over-strong. In fact, both teams cite each other’s work. Is the debate about the timing of who said what to whom? Really?
Kissinger said academic politics are so vicious because the stakes are so small. Yes.
Academics are notoriously ego-driven and they can certainly be petty, but it seems really obtuse in this case to characterize an advisor standing up for his grad student as an issue of petty ego-driven academics.
Exactly. No actual evidence of ideas being stolen. Just jealousy that someone reached print first? That someone didn’t cite their grad student?
Quoting Jay’s first post: “Nor did APW acknowledge that their study was essentially a replication of Jon and Pat’s earlier study. The research designs were nearly identical. The data were almost the same. The only difference was that Jon and Pat had a more complete data set and as a result reported more negative results.” [sic]
Mark, do you think that failure to cite is a form of academic fraud? Especially when it’s a nearly identical research question?
It is not true that APW cited the original research by Wolf/Mills.
Moreover, they didn’t disclose to Wolf/Mills that they were working on a very similar project when consulting with them. That’s not rotten?
APW’s footnote 5 on page 13 cites Mills and Wolf (2016).
I made no assessments of character or behavior. The problem with ‘he said he said’ kind of stuff is that observers cannot know conclusively what was said by whom and when. Jay reports one side, Chris presents another side.
When a researcher fakes data, that’s fraud. When a researcher puts his name on findings generated by another researcher, that’s theft. What I am reading here is not fraud and not theft.
And, what’s the point? Should readers interpret the findings differently? A strong point of randomized trials is that findings are straightforward. It’s about subtracting two averages. The teams then take different approaches in how they explore other aspects of the data. For example, the MIT team looks at quality of schools attended by control group students. UArk looks at whether whether the design of the state test might explain findings. The diversity here is a plus.
If the UArk team decides they definitely are not sending holiday greetings cards to the MIT team, I can understand that.
It is intellectually dishonest to frame a competing paper (which incidentally precedes their own work) as a follow-up and to relegate it to a footnote without talking about the distinction between the two projects.
If that’s considered acceptable behavior, I suppose some people have very low standards of scientific integrity.
It is clear that APW acted badly
It seems like it would have been straightforward and easy for APW to have erred on the side of transparency. Then there would have been their work and that of the UArk team.
The other issue is that one group of researchers has been complaining in the press about losing their data-sharing agreements with the State of Louisiana.
Had Prof. Greene not spoken up, the false implication that they lost access to LA data because of their negative findings may have been allowed to stand.
And that could affect the public perception of ALL research evaluations of school choice programs, which need to remain free of allegations that researchers collude with (or face coercion from) government officials and program administrators.
Did APW steal your data, ideas, methodology? No.
Did APW cite your paper? Yes.
“Oh, but they did not acknowledge us properly, they did not say we were the first.” Cry me a river.
Btw, your initial post created the impression that APW stole the idea from the presentation at the conference, and then they rushed to publish it. Their response, which showed that they have been working on this since 2012, took the wind out of that claim. Now you guys are back to “They did not cite us good enough” argument. Which sounds pathetic.
“The reason that university politics is so vicious is because stakes are so small” Henry Kissinger.
So true…
Keep deflecting and misdirecting instead of engaging Greene.
APW should be sanctioned for violated their data agreement.
Nowhere in the first post did Greene explicitly argue that the idea for the project was stolen. Greene argued that APW unethically rushed out their results in order to deny credit to Mills and Wolf, who had actually completed the work first, and then failed to credit Mills and Wolf for their work. The failure to credit is ongoing; the APW paper’s description of its place in the literature is still fundamentally dishonest, in a completely self-serving way.
Quoting Jay’s first post: “Nor did APW acknowledge that their study was essentially a replication of Jon and Pat’s earlier study. The research designs were nearly identical. The data were almost the same. The only difference was that Jon and Pat had a more complete data set and as a result reported more negative results.” [sic]
Of course he did not explicitly argue that APW stole the idea, but he created that impression. (see the quote below).
When APW shot back, documenting that had been working on this project since 2012, he retracted to the citation issue.
When it comes to shading the truth Greene, it seems, is no better than APW.
I believe that a big part of the blame goes to the people who implemented the program and thought that it was a “good” idea to release the data to two groups to study the same outcomes – what a waste of resources and an unfortunate blame game! As far as “who came up with the idea first” goes, why don’t the two groups release their DUA and the specific aims and approaches they proposed? Release of results first does not necessarily mean the idea was also theirs.
Multiple researchers’ having access to data is a good thing — once you create the dataset, providing it to more researchers is of relatively small added direct cost (more in revising/reviewing the legal agreement than shipping the data). Transparency by sharing data isn’t a problem.
You are missing the point here talking about marginal cost but forgetting about marginal benefit. I specifically mentioned “to study the same outcomes” for a reason. Replication studies are needed in many instances to get to the “truth” but spiraling this competition among academics on “let’s see who has more RAs to get results early” is neither good for academics, nor a sound economic advice! Sharing data once you submit your findings is good and allows for others to verify “outrageous” findings, if any.
I am still waiting on the authors to release their DUAs to settle this issue of “I came first with the idea”!
And it seems to me, Sherman, after wading through too much of this, that the pivotal question is that self-interested education authorities and elected officials get to decide who has access to data and who does not. That is a great deal more important question for the integrity of research than who cited who and when, especially given that work was done independently.
Amen to Leo Casey.
Questions for Leo: Should students not have confidentiality protection for their personal information? If they should, who should decide who gets access to it, if not state departments of education?
Excuse me, should have said “Darth Leo.”
Ah, Greg, always one to elevate the level of debate.
No entities with vested interests in the policies that are being evaluated should not be the bodies that get to decide who is able to do research on the efficacy of those policies. That is textbook self-dealing. It would of course be a simply matter to set up independent review boards that could make such decisions, and perform all the necessary due diligence around issues of confidentiality of records and such. Having spent a year and a half working to extract data on teacher race and ethnicity — data far from any issue of confidential student records — while state boards of education, school districts and charter schools that did not want to be accountable for what was happening on their watch did everything in their power to stonewall the requests, I have had first hand experience is how this can play out. Interesting that you have never experienced it, Greg.
I’ve experienced frustrations getting data that ought to be public as well; I don’t know why you’d assume otherwise. But any data access system you build will be subject to all the same political pressures this one is. Google “regulatory capture.”
On another subject: You’re currently working with teachers at KIPP AMP who want to form a union. Tell us more about that situation. Why did they come to you?
Obviously, if APW’s behavior in this case were acceptable under the ethics of our profession, people like Pat and Jon would not present their work at conferences. That is why the ethics of our profession forbid this behavior, and why this is not a minor violation but one worth taking very seriously.
If APW pay no price for doing this to Pat and Jon, why should future researchers be transparent with their colleagues?
Science is a community enterprise; it advances when we share our work with each other so we can accumulate knowledge and critique each other’s work. The ethical rules APW violated exist to make that process safe for those who have less power. When vulnerable parties are not protected from abuse by the powerful, we all suffer.
The use of power makes science impossible. Science can only progress where people are protected against powerful people’s desire to exclude or exploit them.
I still don’t see what APW did wrong. If they had used an insight they got from the presentation, some tiny shred of intellectual property, without acknowledging the source that could be a problem, but Jay didn’t argue that. He acted like the simple idea of evaluating the impact of the program was itself a novelty whose credit must go to the first one who came up with it. That is nonsense. I think New York City’s teacher pay experiment was studied by three different groups of researchers and none of the researchers was required to pay homage to the others. Jay is teaching his graduate student how to be petty and vindictive.
Quoting Jay’s first post: “Nor did APW acknowledge that their study was essentially a replication of Jon and Pat’s earlier study. The research designs were nearly identical. The data were almost the same. The only difference was that Jon and Pat had a more complete data set and as a result reported more negative results.” [sic]
Take my advice and quit digging.
Greg, the research design isn’t exactly all that novel, period, from either party. It follows pretty straightforwardly from prior work by APW and their research group, so the idea that APW stole the design is more than a bit silly.
I’ve read the quote from Jay’s first post in the original and in the multiple times Greg Forster cut and pasted it. It doesn’t say what was novel about Mills or Wolf’s work.
The novelty was that it happened first. Some throw-away line in APW’s boiler-plate data agreement is not research. Multiple conference presentations and a dissertation by Mills and Wolf is research. When APW present their research in 2015 as new, and do not acknowledge that someone had completed essentially the same research previously, which they were clearly aware of, then they are liars. When they attempt to remedy the lie with a self-serving and still fundamentally dishonest footnote, they are making the lie worse. How is this difficult to understand?
That is ridiculous. The study’s methods — comparing lottery winners to losers — are lifted from straight from the work of Caroline Hoxby (2004) and of Phil Gleason and his co-authors, and many others, with no citation. See here for a long list of people, including APW and colleagues, who pioneered the methods that the Arkansas team used. https://www.povertyactionlab.org/sites/default/files/publications/what-can-we-learn-from-charter-school-lotteries.pdf You don’t see those authors crying about fraud. This race to be first on Louisiana results is harmful to the field. We only care about the weight of the evidence and having multiple teams look at the same data is a good thing.
You are strangely preoccupied with the methodology, even though nobody is making the case that the methodology deserved a citation. The claim is the analysis of the first year of data from the Louisiana program had already been completed prior to APW’s analysis, and they knew about it. The fact that they used essentially the same methodology is also relevant, but it’s not the main point. The main point is it was the essentially same analysis, with the same data and the same findings.
And by the way, Hoxby didn’t invent this either. You’d have to go a bit further back than that. But again, the methodology isn’t the main point, nor could it be. There is nothing particularly novel about the methodology.
The whole point of citing someone else is to give credit for their intellectual contributions that you are using or building on, not to acknowledge that someone is on your turf (or you are on theirs, but it really shouldn’t be anyone’s turf).
No, citations are also necessary so people can evaluate your work in light of other relevant work. That it what it means for there to be no “turf.” The reason our profession has an ethical rule that you must cite relevant work you know of is precisely to prevent the development of “turf.” If APW don’t cite relevant work they know of, they are (functionally, whatever their intentions are) cultivating a “turf.”
Unless I’m missing something here (and I may very well be so correct me if I am), nobody contests that APW knew about Pat and Jon’s work and didn’t cite it. The argument for APW seems to be “it’s no big deal.” That seems to implicitly concede they broke a rule (always cite relevant work that you know about) and argue it’s like crossing against the light when you can see there are no cars coming – it’s technically a violation but there are no consequences so it doesn’t matter.
Of course there was at minimum the consequence to Pat and Jon that readers of APW were not informed of Pat and Jay’s work when they ought to have been. But let me point out that APW’s violation also made it very reasonable to ask the questions Jay has been asking. Does all this resulting hubbub feel like no consequences?
Next time if they follow the rules we can all be spared. This seems to me to adequately illustrate the justice of the rule and the foolishness of breaking it.
So when will the fundamentally dishonest footnote be changed into a proper citation?
Not only is the horse you are so vigorously beating dead; it is an advanced state of decay.
I don’t see a dead horse. I see a forthcoming paper in a respected journal that includes a dishonest footnote instead of a proper literature review. As far as I would say the paper should be retracted on grounds of that.
I do not see it anywhere else here, the paper with the dishonest footnote is forthcoming in American Economic Journal: Applied Economics https://www.aeaweb.org/articles?id=10.1257/app.20160634&&from=f
The recent PhD graduate’s paper ended up in Educational Evaluation and Policy Analysis http://journals.sagepub.com/doi/abs/10.3102/0162373717693108%20
The difference in prestige of the journals most probably cost the recent graduates quite some job opportunities.
You may find it interesting if not amusing that Pathak comes from long lineage of royal priests in Nepal. His grandfathers and great grandfathers were big thinkers, scholars of Sanskrit, and strategists and advisors to multiple generation of Kings of Nepal. They were powerful enough to be assigned their own small battalion of soldiers called “Priest’s Battalion” which Pathak must’ve seen in his Nepal trips. You may know Nepal sits between two behemoths – India & China, and role of such royal scholars and strategists were vital for its survival and operation. So Pathak’s families still enjoy quite a stature back in Nepal even today, although no King rules Nepal anymore. That gene and that brain still clearly lives on!