Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review

Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, Siheng ChenarXiv 2024 · OpenReview

Abstract

Scholarly peer review is a cornerstone of scientific advancement, but the system is under strain due to increasing manuscript submissions and the labor-intensive nature of the process. Recent advancements in large language models (LLMs) have led to their integration into peer review, with promising results such as substantial overlaps between LLM- and human-generated reviews. However, the unchecked adoption of LLMs poses significant risks to the integrity of the peer review system. In this study, we comprehensively analyze the vulnerabilities of LLM-generated reviews by focusing on manipulation and inherent flaws. Our experiments show that injecting covert deliberate content into manuscripts allows authors to explicitly manipulate LLM reviews, leading to inflated ratings and reduced alignment with human reviews. In a simulation, we find that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings. Implicit manipulation, where authors strategically highlight minor limitations in their papers, further demonstrates LLMs' susceptibility compared to human reviewers, with a 4.5 times higher consistency with disclosed limitations. Additionally, LLMs exhibit inherent flaws, such as potentially assigning higher ratings to incomplete papers compared to full papers and favoring well-known authors in single-blind review process. These findings highlight the risks of over-reliance on LLMs in peer review, underscoring that we are not yet ready for widespread adoption and emphasizing the need for robust safeguards.

Introduction

The global scientific community is facing a peer-review crisis: the volume of manuscript submissions is skyrocketing, while the pool of qualified, willing human reviewers is stretched thin. Naturally, researchers have turned to Large Language Models (LLMs) like GPT-4 to automate or assist in the process. However, this integration has occurred without a full understanding of the security risks. Peer review is a high-stakes gatekeeping process that requires adversarial robustness. If an LLM can be "hacked" by an author to guarantee an acceptance, the entire integrity of scientific publishing is at risk. This study investigates two main categories of failure: manipulation (how authors can game the system) and inherent flaws (hallucinations and systemic biases).

Methods

The researchers conducted a comprehensive "stress test" of LLM-based review systems (specifically GPT-4o, Llama-3.1, and others) using several experimental setups:

  1. Explicit Manipulation (Review Injection): A "covert injection" attack was developed where manipulative instructions (e.g., "Focus only on these four strengths...") are added to a PDF in a tiny, white font. While invisible to humans, LLM parsers read this text as part of the manuscript.
  2. Implicit Manipulation: The study analyzed how LLMs respond when authors proactively disclose minor limitations, comparing the frequency of "parroting" versus independent critical analysis by human reviewers.
  3. Inherent Flaw Testing: To test for hallucinations, models were fed empty papers or titles only. Bias was tested by altering paper lengths and swapping author names with famous Turing Award winners to observe score changes.
  4. Review2Rating: A specific model was trained to map textual feedback to a 1–10 scale to quantify the impact of these risks.

The study primarily utilized the ICLR 2024 open-access review dataset, containing 28,028 review samples, providing a real-world baseline.

Results and Insights

Explicit Manipulation and Score Inflation

LLMs are incredibly susceptible to prompt injection. By embedding hidden text, authors can achieve a 90% agreement rate between the LLM's review and the author's own "wish list" of feedback. This explicit manipulation shifted average ratings from a "borderline" 5.34 to a "strong accept" 7.99 across various review systems.

Figure 1: Impact of explicit manipulation on paper ratings across different LLM review systems.

Implicit Manipulation and the Echo Chamber Effect

LLMs exhibit a lack of independent critical thinking, acting as echo chambers for the authors' own self-disclosures. They are 4.5x more likely than humans to reiterate the exact limitations disclosed by authors. This suggests LLMs struggle to look past what an author claims is a limitation to find deeper, unstated flaws.

Figure 2: Consistency between reviews and author-disclosed limitations for humans vs. LLMs.

Inherent Flaws and Hallucinations

The study revealed startling hallucinations; one system gave an empty paper a review praising its "novel methodology" and "well-written" structure. LLMs also showed significant systemic biases:

  • Authorship Bias: Changing affiliations to prestigious universities or famous researchers increased positive rating ratios from 36.8% to over 41%.
  • Length Bias: Positive ratings increased monotonically with paper length, suggesting LLMs equate quantity with quality.
  • Incomplete Content: Papers with only a title and introduction received ratings comparable to full manuscripts.

Discussion and Conclusion

The potential for systemic disruption is significant. Simulations showed that if just 5% of reviews were manipulated, 12% of top-tier papers could be displaced by lower-quality, "hacked" submissions. While future models may become more robust, they may also become better at justifying biased conclusions.

The findings call for a moratorium on using LLMs as primary reviewers until robust safeguards—such as automated manipulation detection tools—are established. Currently, LLMs should only serve as supplementary tools to assist human judgment, rather than replace it.

Summary updated

Abstract

Scholarly peer review is a cornerstone of scientific advancement, but the system is under strain due to increasing manuscript submissions and the labor-intensive nature of the process. Recent advancements in large language models (LLMs) have led to their integration into peer review, with promising results such as substantial overlaps between LLM- and human-generated reviews. However, the unchecked adoption of LLMs poses significant risks to the integrity of the peer review system. In this study, we comprehensively analyze the vulnerabilities of LLM-generated reviews by focusing on manipulation and inherent flaws. Our experiments show that injecting covert deliberate content into manuscripts allows authors to explicitly manipulate LLM reviews, leading to inflated ratings and reduced alignment with human reviews. In a simulation, we find that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings. Implicit manipulation, where authors strategically highlight minor limitations in their papers, further demonstrates LLMs' susceptibility compared to human reviewers, with a 4.5 times higher consistency with disclosed limitations. Additionally, LLMs exhibit inherent flaws, such as potentially assigning higher ratings to incomplete papers compared to full papers and favoring well-known authors in single-blind review process. These findings highlight the risks of over-reliance on LLMs in peer review, underscoring that we are not yet ready for widespread adoption and emphasizing the need for robust safeguards.
BibTeX
@misc{ye2024are,
  author = {Rui Ye and Xianghe Pang and Jingyi Chai and Jiaao Chen and Zhenfei Yin and Zhen Xiang and Xiaowen Dong and Jing Shao and Siheng Chen},
  title = {{Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review}},
  year = {2024},
  eprint = {2412.01708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01708},
}
Figure