Peer Review: “The Worst Way to Judge Research, Except for All the Others”
Peer review asks other experts to assess research before publication. It can identify weaknesses and improve a paper, but reviewer bias, uneven effort, and limited editorial resources mean that publication does not guarantee validity.
THE IDEA
Peer Review: “The Worst Way to Judge Research, Except for All the Others”
We wish we could say that every article that makes its way through the gauntlet of peer review is printworthy. No system is perfect, and peer review is no exception. Peer review runs on the backs of unpaid volunteers who have agreed to serve as the guardians of quality control in their respective disciplines. Some reviewers take their job seriously and write detailed and helpful reviews. Others dash off perfunctory comments barbed with stinging insults. There’s even a whole website where disgruntled authors post the nasty things reviewers have written.[3] And let’s not forget about the biases and prejudices that creep in whenever humans express judgment. Many journals try to compensate for these foibles by following a “double-blind” procedure: reviewers don’t know who the authors are and vice versa. But experts often have a sense of who is doing what in their field, and the pretense of anonymity turns out to be a hollow shell game.
For some journals, like the prestigious New England Journal of Medicine, peer review is only the first stage in a rigorous publication process. If reviewers agree that a manuscript is of sufficient merit, it’s forwarded to a group of in-house editors who go through it with a fine-tooth comb. But the New England Journal of Medicine is the exception. Only a dozen or so of the ten thousand biomedical journals can afford a staff of paid editors.[4] The vast majority rely on the kindness of volunteers. When the system works, it does a decent job of separating good scholarship from bad. When it doesn’t, slipshod work finds its way into print. Yet, despite its many problems, the tendency of peer review to produce better results on average than other methods has spurred many of the advances we see today. An article by an editor at the Journal of the American Medical Association put it like this: “Peer Review: The Worst Way to Judge Research, Except for All of the Others.”[5]
When to use it
Evaluating a research paper submitted to an academic journal for publication.
Reviewing grant proposals to fund scientific research.
Assessing the validity of a medical study before its results are applied in clinical practice.
Deciding whether to accept a conference presentation based on submitted abstracts.
FROM THEORY TO EVERYDAY LIFE
In practice
Flaws in the Peer Review Process
The authors describe uneven effort among volunteer reviewers: some offer detailed feedback, while others give perfunctory or insulting comments. Double-blind review attempts to reduce some biases, although specialists may still infer authorship.
Exception: Top Journals with Paid Editors
The authors describe the New England Journal of Medicine as adding in-house editorial review after external peer review. Their account estimates that only about a dozen of 10,000 biomedical journals could afford comparable resources; this is the book’s estimate rather than a current journal census.
A MOMENT TO REFLECT
Put it to work.
How can peer review be redesigned to minimize the impact of reviewer biases?
What are the ethical obligations of reviewers who volunteer their time?
In what scenarios might double-blind review actually be less effective than single-blind or open review?
CONTINUE THE CONVERSATION
Source texts
- Verified How to Think Straight, Get Duped Less, and Make Better Decisions about What to Believe Online (Mike Caulfield, Sam Wineburg)