Peer Review: Definitive Truth or Suboptimal Standard? Four Views

The peer review process has long handled the role of academia’s “referee” for determining the legitimacy of research. Many academics say it has served extremely well, yet ever since the Sokal Hoax in 1996, in which a leading peer-reviewed journal published a bogus article, peer review’s central position in the world of knowledge has increasingly raised eyebrows.

In some cases, particularly in the humanities, it has been criticized as a system of filtering out ideas that do not adhere to a specific ideology. In the natural sciences or the social sciences, there have been several recent scandals, such as the retraction of 43 papers with fabricated reviews by BioMed Central, or Michael LaCour’s falsification of data concerning attitudes toward gay marriage. There have also been well-known intentional tests of the peer review, such as the British Medical Journal deliberately inserting eight errors into an unpublished paper and reviewers uncovering on average only two of them.

And yet, there appear to be few alternatives on the horizon, and many scientists and scholars support peer review. Perhaps the peer review process is fine as it is, or it merely needs some minor tweaking to continue performing its central role in the intellectual world. Or, perhaps, it needs major changes.

The Pope Center asked four distinguished academics who have been involved with the peer review process as editors, participants, or critics for their opinions. Here are their responses.

The System Works 

By Tom Birkland

Like all human endeavors, peer review is not perfect, but it’s also true that “the system works.” The most important studies—those that gain the most attention, or that have counter-intuitive results—are the ones that receive the most scrutiny. This is how science should work. Peer review is but one aspect of this process, and, while its value has been rightly questioned in recent years, it is hard to imagine an alternative that would be as successful in ensuring that the best science is published.

There seems to be some confusion among critics about what peer review does and does not do. The idea of peer review is to ensure that a scholarly work—usually an article in a scholarly journal—has addressed an important question and used appropriate methods and sources to reach a plausible conclusion.

Peer-review is not about accuracy as much as it is about plausibility, because our role as peer reviewers is to ensure that the work is true to the scientific method, however that’s defined. Peer review is often less problematic in the natural sciences than in the social sciences because, in the natural sciences, a phenomenon will occur exactly the same way every time, given the same conditions. It is therefore easier to “settle” controversial questions through empirical experimentation and observation. The social sciences are subject to the entire range of mankind’s complex thought processes and emotions; therefore, it is more difficult to settle controversial questions through experimentation.

The humanities also have peer-reviewed journals, although the standards of proof and analysis in the humanities differ somewhat from the natural and social sciences, because humanistic work is more grounded in criticism and analysis than in empirical testing of theories.

Something peer reviewers do not do is replicate studies. Instead, it’s up to the scientific community to replicate and investigate the ideas revealed in a published study. For that reason, the National Science Foundation, which funds considerable research, now requires “data management plans” to allow researchers to gain access to data. A major political science journal has recently announced that it will not publish studies without data that can be used for replication. Other disciplines are taking a much harder look at replication.

Of course, problems exist within the peer review system. There have been some notorious examples of misconduct in recent years, such as the recent research on social acceptance of gays and lesbians that was withdrawn when the senior author learned that the junior author of the paper, a graduate student, falsified data. The natural sciences are not immune to data fraud, as revealed by “cold fusion” and “vaccines and autism” fiascos in the 1990s.  In all three of these cases, failures to replicate the original study revealed the work, in those cases, to be fraudulent, and the result is that the original papers in those cases are not considered part of the scientific literature.

While spectacular cases of research misconduct, or even just really poor methods, can and should cause us to think about our processes, these cases are also unrepresentative of the way most scholars conduct and assess their peers’ research. Indeed, to accept such cases as typical would be as scientifically invalid as some of the most notorious frauds.

The remarkable success of all the sciences in the past sixty years suggests that our system of peer-review works well, and that reforms should be oriented to improving peer review, not replacing it with unproven methods.

Spurring Groupthink 

By Barbara Oakley 

I don’t have a dog in the fight of climate science. I’m not a climatologist myself—I don’t even play one on TV. I’m personally concerned about the environment, and certainly wouldn’t want to do anything that might harm the world we live in. In other words, I’m in tune with the arguments of climate change proponents.

So it was a shock for me when I was having lunch several years ago with a tight-knit research group at a top university—the group included one of the world’s leading statisticians. As our freewheeling conversation unfolded, the topic turned to climate change. Suddenly, the conversation grew quieter. I noticed furtive glances over the shoulder—obvious attempts to see who might be listening.

“Actually,” the group’s leader noted, “our statistician took a look at a number of the key papers underpinning the idea of global warming.” He looked meaningfully towards the statistician.

There was a pregnant pause. Then the statistician said, sotto voce—“there were so many amateur mistakes in those papers that it was hard to believe that anyone could take them seriously.”

There was a moment of silence around the table. “Why don’t you publish something about that?” I asked.

The statistician looked at me and smiled ruefully. “Because if I did publish, my career would be ruined.”

Science is a human activity, fraught with human frailties. It is predicated upon a belief system that involves the search for objective facts. It also involves the ability to make inferences and to construct models that can allow us to predict new facts. This is notwithstanding the idea that objectivity is itself sometimes hard, if not impossible, to achieve. Moreover, facts themselves can sometimes be legitimately disputed. On top of all that, inferences and predictive models add an entirely new layer of complexity.

It is for these reasons, when we are pursuing science, we often need the feedback of others to make sure we aren’t fooling ourselves. As Nobel Prize winning scientist Richard Feynman once noted: “The first principle is that you must not fool yourself—and you are the easiest person to fool.”

I personally have benefited greatly from the feedback I’ve received through the peer review process. But I’ve also suffered the indignity of having a well-constructed paper come back in a body bag because it did not fit the preconceived notions of the reviewer.

What is often forgotten in peer review are the extraordinary effects of groupthink in rear-ending the scientific process. Perhaps surprisingly, good intentions by scientists—“altruism bias”—can be even more harmful to the scientific process. Such bias not only subverts the search for scientific truth, but also can motivate researchers to attack other researchers who aren’t following the well-intentioned herd. All this helpful groupthink can lead to a false consensus that can fool not only fellow scientists, but the public as a whole.

Imperfect But Essential 

By Joann Keyton

The peer review process is not perfect, but for interdisciplinary and international journals, like the one I co-edit, it is essential. In many ways, peer review helps to establish an open market place for scholarship.

Generally, the peer review process works like this: An author submits anonymously to a journal, the editor assigns the submission to two or more reviewers; when those reviews are returned, the editor evaluates the submission, makes a decision (accept, revise & resubmit, or reject), and writes a response letter including the reviewers’ comments) to the author.

At its best, double-blind peer review is a three-way collaborative learning process among author, reviewers, and the editor with the goal of improving the research published in scholarly journals. And, while editors appear to hold ultimate power by deciding what is published or not, an editor can only publish from among manuscripts submitted to the journal.

The peer review process for scholarly journals is not standardized within or across disciplines. Certain traditions are invoked, and even, sometimes, protected. Despite objections raised, the peer review process adds value to scholarly journal publishing by raising questions about theory and methods, and helping authors see their theory and data from a less limited point of view.

Like democracy, people have differing points of view, and it is the editor’s responsibility to engage the parties without exposing their identities, and to make a final editorial decision. So, like democracy, people get to express their opinions, but the conclusion may not be satisfying to every party.

From my point of view, the value of the reviewers’ comments are more important than the reviewers’ recommendations (accept, revise & resubmit, reject) about publication. Some reviewers seldom or never recommend publication. But reviewers’ descriptive and evaluative comments guide me as I read the submitted manuscript and evaluate its strengths and weaknesses, its fit with the journal, and its potential contribution. It is the editor’s role to move the evaluation of a manuscript to a close (accept or reject) or to continue the conversation by asking the authors to use the reviewers’ comments as they revise and resubmit the manuscript.

The double-blind peer review could be improved in three ways. First, larger editorial boards increase the likelihood that the editor can identify expertise-appropriate reviewers. Second, university, college, and departmental evaluation standards should identify and positively reward faculty for their editorial board and review activities. Third, scholars must commit to and uphold the ideal of collaborative learning in the review process. Everyone gains something when learning, rather than judgment, is the motivation for participating in the double-blind review process.

Biased Against Real Discovery, But with No Replacement in Sight 

By John Staddon

Peer review has its problems, but they are part of, and contribute to a larger problem. The social structure for basic-science research has changed enormously since World War II.  Science is now for most scientists a career, not a vocation. A vocation is an activity to be carried on at leisure, guided only by the researcher’s curiosity. But a career means a salary, which depends on the “success” of your research. Success now depends not on your own judgment but on some kind of product judged by others. You cannot afford to waste time and effort pursuing leads that don’t pay off within a couple of years. And “pay off” means just two things: research grants, and publications in the right “peer-reviewed” journals.

As support for science has increased, so has the number of scientists. When I started out, Ph.D.s were few and employment readily available. Applying for federal grants was simple—my first application was about 10 double-spaced pages—and the chances of success high. There was no need to spell out every detail of the proposed research in advance, nor was this possible in such a short space.

Research applications now amount to a sort of science fiction—describing proposed work (often completed but not published) in unrealistic detail. The number of scientists has vastly increased, research is much more expensive, and competition is fierce. No working scientist can afford to offend his peers, who may review his next paper (anonymously) or grant application (almost anonymously). This has led inexorably to two things: less criticism all round, and a search to find a niche in which one can be one of a handful of mutually supporting dominant players. Or perhaps to join a large research group, protected by its size. Consequently the kind of wide-ranging, independent thinking that leads to breakthroughs is now rarer than it used to be.

Still, it’s hard to see an alternative to review by experts in the field. Some kind of review is obviously necessary to filter an otherwise unmanageable number of submissions. An incidental, but scientifically irrelevant, benefit of peer review is that it provides credentialing: high prestige—high “impact”—journals, with a low acceptance rate, count more towards your CV than publication in some online “we accept everything” journal. But it’s the oversupply of papers rather than the filter that is the main problem, I think.

Peer review cannot catch fraud. Science ultimately depends on honesty and trust, both of which are eroded by current incentives. The reviewer does not have the time, resources or incentive to repeat the research he is reviewing, which would be the only real check. A reviewer can judge if an argument is logical and clear, if the method described is capable of yielding the results presented, and if citations are relevant and reasonably complete. The latter is especially important—an old friend of mine recently admitted that the first thing he checked in the Word file of my new book was his own name (he wasn’t happy with the result!). All this favors highly specialized niche research that doesn’t deviate too far from what is generally accepted.

Another problem is that career incentives have favored statistical methods—used to justify many drug and social science studies—that guarantee a certain number of publishable results even if every experiment is really a failure. Now that people are paying some attention, at least in important areas like drug research, it turns out that many studies cannot in fact be replicated.

In other words, the whole system, a system of which peer review is just a part, is biased against real discovery. That’s the problem.