Peer review, one of the foundations of modern science, is a safeguard or quality control used in advancing science. It is an important part of scientific publishing, and it’s often assumed that peer review publications are synonymous with high quality. The use of peer review processes goes back at least 200 years (Kronick 1990).Jefferson and colleagues (2002) claim there are two principal functions of peer review: filtering out incorrect information and improving the accuracy and clarity of scientific reports.
When writing papers for school assignments, students are often told their papers must include citations from peer-reviewed publications. It is implied that if the publication is peer reviewed it is high quality having passed the peer review process.
But what does it mean to be peer reviewed? What are the limitations regarding peer review? Can peer reviewed be improved? Peer review is not defined as a specific step-by-step process used to produce an outcome, and procedures vary tremendously. The value of peer review is often exaggerated and not clearly explained. Procedures have been suggested by researchers that may lead to improvements in peer review quality. Peer review is a gatekeeper for scientific publication and should be used in a way that positively contributes to the scientific literature.
In the peer review process, a paper is submitted to a journal and evaluated by several reviewers (often reviewers are individuals with an impressive history of work in a specific area related to the material being reviewed). Theoretically, peers review a manuscript and offer constructive criticism (Gerwing et al. 2020). In Single Blind Reviews authors do not know who the reviewers are. In Double Blind Reviews authors do not know who the reviewers are, nor do reviewers know the identity of the authors. After critiquing the paper, the reviewers submit their critiques to the editor. Then, based on the commentaries from the reviewers, the editor makes editorial decisions including whether or not the paper is a good fit for the publication. Suggested revisions are sent to the author, and the author is usually expected to address these revisions in a letter sent back to the editor. It isn’t always necessary to make all the changes suggested by the reviewers to be published. However, it is advisable to address each of the comments or suggested revisions in a letter responding to revision requests.
The goals of peer review are to ensure the credibility and integrity of the scientific record by pointing out weaknesses, offering feedback for improvement, and ensuring that misleading science is not published. “Peer review is one way (replication is another) science institutionalizes the attitudes of objectivity and public criticism. Ideas and experimentation undergo a honing process in which they are submitted to other critical minds for evaluation. Ideas that survive this critical process have begun to meet the criterion of public verifiability” (Stanovich, 2007, p. 12).
Peer Review Limitations
Peer review has attracted criticism surrounding issues of an excessively long publication process, bias, and fairness (Schwartz and Zamboanga 2009). While favored by some researchers, other researchers claim we need more evidence to support the idea that the peer review process adds value to papers (Jefferson et al. 2006). Peer-reviewer comments are sometimes unprofessional and involve comments demeaning authors. Another criticism is there is too much focus on variables that are not direct reflections of the technical merit of the submission.
Who are the peers doing the reviews? Does each reviewer have a similar levels of knowledge? Are the reviewers doing the same kind of research (in which case the reviewer may be a direct competitor) as the research that is under review? Do the reviewers work in the same scientific domain? Do they have exceptional knowledge in methodology or statistics? Do the reviewers have specific criteria in determining whether or not paper is published? Are statistical analyses repeated? Are all the references checked? Are clear, detailed suggestions made for improvement?
Andrew Gelman (2016) claims that main problem with peer review is the reviewers. Peer review is subject to groupthink, and there are sometimes incentives to publish things that the reviewers agree with or are already working on. If an entire group of peers has a misconception or they lack knowledge to detect problems in the submitted work, they may accept the work for publication. In such a case, peer review can perpetuate error. It isn’t unusual for peers to lack knowledge in the area they are reviewing. Finding reviewers is difficult and journals may use reviewers based on characteristics other than their knowledge in relevant areas (for example, availability or willingness to offer review).
Publication bias is common in the peer-reviewed literature (Goldbeck-Wood 1999). Publication bias is the tendency to publish studies that show statistical significance while not publishing even high quality studies that do not find statistical significance. A review of twelve antidepressants found that of the studies submitted to the United States Food and Drug Administration that during the approval process, the majority of non-significant results were published less frequently or never published (Turner et al. 2008). A study conducted by Emerson et al. (2010) to determine if positive-outcome bias was present during peer review was interested in whether peer reviewers would “recommend publication of a ‘positive version’ [statistically significant] of a fabricated manuscript over an otherwise identical ‘no difference’ version.” The results indicate reviewers were more likely to recommend publication of the positive version, even though the papers were almost identical, with the only difference being the direction of the finding.
Converging Evidence
Peer review is a major component of scientific publishing, and efforts should be made to improve the process. Such improvements could include: using reviewers that are highly knowledgeable in the areas being reviewed; offering reviewers incentives; speeding up the process and promoting clear, effective communication between reviewers and authors; decreasing the rate of publication bias; approving papers for publication based solely on the merits of the paper (promotion of a consistent meritocracy); requiring specific criteria for publication and requiring specific feedback from reviewers to authors; open publication of reviewer comments; involving statisticians in the review process (many retracted papers are due to statistical errors); and so on.
Converging evidence is the highest level of scientific evidence. It consists of a convergence of evidence involving different researchers, various studies and models, and the use of different methods. When evaluating scientific data, an array of factors needed to be considered in addition to whether it is published in a peer reviewed journal, including: funding sources, study replication, study design, sample size, conflicting interest, sampling error, different measures of reliability and validity, reporting limitations, and other possible criticisms of the study.
There are good studies that do not get published in peer review publications, and there are low quality studies in the peer review literature. The retraction watch database (database listing studies that have been retracted from scholarly journals) includes thousands of studies; many of those studies were published in prestigious peer reviewed journals. It is a mistake to label a study, review, commentary, meta-analysis, or any other paper as high quality based solely on peer review status.
References
Emerson, G.B. et al. 2010. Testing for the Presence of Positive-Outcome Bias in Peer Review. Arch. Intern. Med. 170(21), 1934–1939.
Gelman, A. 2016. When Does Peer Review Make No Damn Sense. Statistical Modeling, Causal Inference and Social Science. Retrieved on November 20, 2020 from https://statmodeling.stat.columbia.edu/2016/02/01/peer-review-make-no-damn-sense/.
Gerwing, T.G., et al. 2020. Quantifying professionalism in peer review. Research Integrity and Peer Review, 5. doi:10.1186/s41073-020-00096-x.
Goldbeck-Wood. S. 1999. Evidence on peer review—scientific quality control or smokescreen? BMJ, 318(7175), 44–45.
Kronick,D.A. 1990. Peer-review in 18th-century scientific journalism.JAMA, 263, 1321–1322.
Jefferson, T., et al. 2006. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews, (1). 10.1002/14651858.MR000016.pub2.
Jefferson, T., et al. 2002. Effects of Editorial Peer Review: A Systematic Review. JAMA, 287(21), 2784–2786. doi:10.1001/jama.287.21.2784.
Schwartz, S.J, and Zamboanga B.L. 2009. The peer-review and editorial system: ways to fix something that might be broken.Perspectives on Psychological Science, 4(1), 54–61. doi: 10.1111/j.1745-6924.2009.01106.x.
Stanovich, K. 2007. How To Think Straight About Psychology 8th edition. New York, NY: Pearson.
Turner, E.H., et al. 2008. Selective Publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358(3), 252–260.
