The Ethics of Peer Review: Problems (2/5)
(May 11th, 2016) Peer review is at the heart of research communication. However, the process is far from perfect and many problems have been identified. Among them, ethical questions about the responsible conduct of reviewers who can wield considerable power behind a cloak of anonymity.
A journal editor’s perspective on problems with peer review
Since journal editors were instrumental in implementing the peer review process as we know it (see Part 1), it is interesting to hear their perspective on the problems and possible solutions. Richard Smith was the editor of BMJ (the British Medical Journal) and chief executive of the BMJ publishing group for 13 years; in his article “Peer review: a flawed process at the heart of science and journals”, he describes the problems he encountered with peer reviewing and the various pioneering experiments that BMJ tested to improve the process.
He points to many negative aspects of the peer review system. “We have little evidence on the effectiveness of peer review, but we have considerable evidence on its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud, it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused.”
Smith says the inconsistency can be laughable. “Here is an example of two reviewers commenting on the same papers.
Reviewer A: ‘I found this paper an extremely muddled paper with a large number of deficits’
Reviewer B: ‘It is written in a clear style and would be understood by any reader’.”
This kind of inconsistency can make peer review like a “lottery”. When you submit a study to a journal, it enters a system that is effectively a “black box”, and then a more or less sensible answer comes out at the other end. “The black box is like the roulette wheel, and the prizes and the losses can be big. For an academic, publication in a major journal like Nature or Cell is to win the jackpot.”
Smith suggests possible improvements, such as standardising procedures; blinding reviewers to the identity of authors; training reviewers; being more rigorous in selecting and deselecting reviewers; or even creating “professional review agencies”. But to what extent can authors trust peer reviewers to provide fair, honest and competent reviews? As he says, reviewing is meant to be objective, but inevitably it involves a great many subjective and inconsistent judgements: “The evidence is that if reviewers are asked to give an opinion on whether or not a paper should be published they agree only slightly more than they would be expected to agree by chance.”
Nevertheless, Smith points to an assumption of trust underlying the whole peer review system (and indeed science as a whole). When reviewing a manuscript, journal editors and peer reviewers trust the authors’ scientific honesty and integrity. They assume that any errors or defects in the manuscript (or the work it describes) are due to innocent mistakes and misunderstandings. But what happens if they suspect that data is fraudulent, or that it has been unethically manipulated? Should peer review continue to operate on trust or should it move more towards the "world of audit"? Smith says some journals, including the BMJ, now make it a condition of submission that the editors can ask for the raw data behind a study. However, he admits it doesn’t work very well: “We did so once or twice, only to discover that reviewing raw data is difficult, expensive, and time consuming.”
After all his years as a journal editor, Richard Smith concludes that “peer review is a flawed process, full of easily identified defects with little evidence that it works. Nevertheless, it is likely to remain central to science and journals because there is no obvious alternative, and scientists and editors have a continuing belief in peer review.”
The authors’ perspective
So how do the researchers who write the manuscripts feel when confronted with the peer review process? Three common problems tend to crop up:
1. Reviewers condemn an article for spurious or incorrect reasons. On the one hand, this may be because they are lazy or negligent, for example, they have not read it correctly or made any effort to understand the contents. On the other hand, there are suspicions that reviewers (under cover of anonymity) may be taking the opportunity to personally attack a rival or a past enemy.
2. Reviewers demand additional experiments that they claim are necessary before the manuscript will be acceptable for publication. Often authors will consider such experiments to be unnecessary or unhelpful for the work in question or to represent a volume of work at least equivalent to that which has already been invested in the manuscript, that is, extra work that might better be spent on research for the next manuscript. Again, reviewers may be seeking to delay or hinder competitors with such tactics, but equally it may be a careless and thoughtless request by a reviewer who hasn't thought through the consequences of what they are demanding.
3. Finally, researchers who receive negative reviews may object that other articles that have been published in the same journal appear to be obviously flawed and substandard. Why have they been published when our manuscript is better? Where’s the fairness and objectivity in the reviewing process? In these circumstances, the reviewers and journal editors are held to be at fault for not being stringent enough in their reading and evaluation of manuscripts. Once again, this may be due to laziness and careless partial readings, but there are also cases where the prominence of authors and their home institutions have influenced these decisions.
Clearly, peer review of scientific articles (and indeed of job or grant applications) poses many ethical questions. Part 3 looks at attempts to define the ethics of peer review and oaths for peer reviewers.