The Ethics of Peer Review: ‘Open’ Reviewing (4/5)
(May 17th, 2016) Peer review is at the heart of research communication. However, the process is far from perfect and many problems have been identified. Among them, ethical questions about the responsible conduct of reviewers who can wield considerable power behind a cloak of anonymity.
There have been calls to clearly define the ethical responsibilities of reviewers and to encourage their adherence to oaths (see Part 3). Others have called for reforms of the entire peer review process to address its failings (see Part 2). Here, we will consider a combination of both these approaches – ethical oaths that promote changes in the way peer review is conducted.
“A letter from the frustrated author of a journal paper”
This was the title of a satirical editorial in The Journal of Systems and Software on the potentially explosive tensions within authors when confronted with what they perceive to be abusive peer reviewers. In the letter, the ‘frustrated author’ is re-submitting the “re-re-re-revised revision of our paper… We have again rewritten the entire manuscript from start to finish… Hopefully, we have suffered enough now to satisfy even you and the bloodthirsty reviewers. Because it is fairly clear that your anonymous reviewers are less interested in the details of scientific procedure than in working out their personality problems and sexual frustrations by seeking some kind of demented glee in the sadistic and arbitrary exercise of tyrannical power over hapless authors like ourselves who happen to fall into their clutches.”
Nevertheless, when considering “this batch of reviewers, ‘C’ was clearly the most hostile, and we request that you not ask him to review this revision. Indeed, we have mailed letter bombs to four or five people we suspected of being reviewer ‘C’, so if you send the manuscript back to them, the review process could be unduly delayed.” The ‘frustrated author’s’ satirical letter continues in similar vein, combining insults and threats against the journal editor and reviewers. Fortunately, the extreme frustrations of real authors rarely result in such violent retaliations against reviewers! Instead, some frustrated authors have called for major changes to the peer review process in order to avoid any further aggravation.
‘The Reviewer's Oath’
Bioinformatician Mick Watson proposed ‘The Reviewer's Oath’ in reaction to a really bad review of one of his papers: “I was unfortunate to receive this morning a negative review of one of the papers we have under review. This review is terrible! It contains so many factual inaccuracies that the reviewer cannot possibly have read the paper, our software tutorial, the code nor the documentation. It contains ridiculous, unsupported conjecture. It is just a really terrible review, and I can only conclude the reviewer is trying to block the publication of our paper.” He laments that he and his co-authors spent months working on this paper, and this is the review they received. “Words fail me! Don’t they deserve more?!”
Watson says that we ought to reverse the whole perspective of peer reviewing. Instead of criticising manuscripts and looking for negative points, maybe it would be better if reviewers took a more positive attitude and asked themselves how they might help the authors to improve their manuscript. But to achieve this, he says, we need to know the identity of the reviewers. “Frankly, reviews that are simply designed to block publication of papers border on scientific misconduct, and those who give them should be publicly identified. Removing anonymity is important! Open review is important! It gives the reviewer a different perspective. I know, because I practice it. It makes you more constructive. It makes you think ‘How can I help?’ rather than ‘How can I criticise?’”
Therefore, he proposes that reviewers should reveal their identity to authors and sign up to his 4-point “reviewer’s oath”:
“I, the reviewer, promise:
i) to not hide behind a screen of anonymity
ii) to be open and honest with you (the authors) at all times
iii) to be constructive in my criticism
iv) within the rules given to me by the journal, to assist you in every way I ethically can to get your manuscript published, by providing criticism and praise that is valid and relevant.”
The Open Peer Review Oath – Ambassadors for Open Science
Open science is a movement that seeks to ensure that the results and the data of scientific research are, and continue to be, available to all. In the paper calling for “An Open Science Peer Review Oath”, the authors explain how they met at the 2014 ‘AllBio’ workshop on ‘Open Science & Reproducibility Best Practice’ and discussed ways of using ethics to promote open science practices. They propose a peer-review oath and accompanying manifesto to help increase the transparency of the scientific method and the reproducibility of research results. These have been designed to offer guidelines to enable reviewers (with the minimum friction or bias) to follow and apply open science principles. “Introducing the oath and manifesto at the stage of peer review will help to check that the research being published includes everything that other researchers would need to successfully repeat the work.”
Since peer review is the lynchpin of the publishing system, they say that encouraging the community to consciously (and conscientiously) uphold these principles should help to improve published papers, increase confidence in the reproducibility of the work and, ultimately, provide strategic benefits to authors and their institutions. “Importantly, peer review happens at a personal, rather than institutional, level and is carried out by individuals; it is therefore an ideal mechanism for engaging most researchers, given that all scientists peer review or are peer reviewed.”
In practice, the authors looked at how they could build upon the existing peer review oaths (including the Peer Reviewer’s Oath (see Part 3), the Reviewer's Oath (above), and ‘My Reviewer Oath’.
Their main amendment has been through the addition of a new declaration, “I will be an ambassador for open science” and they recommend that this should be included in review preambles. For it to be successful, they note that reviewers should “follow the spirit of the statement throughout their reviews”, even though it may involve a little additional work. The full publication history of their article, reviewers comments (and identities) are provided online. In their first version, they presented a 17 point oath together with a manifesto. In his own open review, Jonathan Eisen (an editor of the open access journal, PLOS Biology) explained that their oath was far too long. After heavy revision, the final version now contains just four principles:
Principle 1: I will sign my name to my review
Principle 2: I will review with integrity
Principle 3: I will treat the review as a discourse with you; in particular, I will provide constructive criticism
Principle 4: I will be an ambassador for the practice of open science
The final version of “A Manifesto with Guidelines for open science reviewers” makes 8 points that closely follow those expressed in the previous oaths, starting with – “I will work with you to help improve your research, as I believe that peer review should be an open, supportive and collaborative process. I will therefore sign my review and state my identity.”
The final part of this series of articles looks at how the principles of open science may signal the end of the traditional peer review process.