Abstract
Scholarly peer review is a cornerstone of scientific advancement, but the system is under strain due to increasing manuscript submissions and the labor-intensive nature of the process. Recent advancements in large language models (LLMs) have led to their integration into peer review, with promising results such as substantial overlaps between LLM- and human-generated reviews. However, the unchecked adoption of LLMs poses significant risks to the integrity of the peer review system. In this study, we comprehensively analyze the vulnerabilities of LLM-generated reviews by focusing on manipulation and inherent flaws. Our experiments show that injecting covert deliberate content into manuscripts allows authors to explicitly manipulate LLM reviews, leading to inflated ratings and reduced alignment with human reviews. In a simulation, we find that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings. Implicit manipulation, where authors strategically highlight minor limitations in their papers, further demonstrates LLMs' susceptibility compared to human reviewers, with a 4.5 times higher consistency with disclosed limitations. Additionally, LLMs exhibit inherent flaws, such as potentially assigning higher ratings to incomplete papers compared to full papers and favoring well-known authors in single-blind review process. These findings highlight the risks of over-reliance on LLMs in peer review, underscoring that we are not yet ready for widespread adoption and emphasizing the need for robust safeguards.
Introduction
The global scientific community is facing a peer-review crisis: the volume of manuscript submissions is skyrocketing, while the pool of qualified, willing human reviewers is stretched thin. Naturally, researchers have turned to Large Language Models (LLMs) like GPT-4 to automate or assist in the process. However, this integration has occurred without a full understanding of the security risks. Peer review is a high-stakes gatekeeping process that requires adversarial robustness. If an LLM can be "hacked" by an author to guarantee an acceptance, the entire integrity of scientific publishing is at risk. This study investigates two main categories of failure: manipulation (how authors can game the system) and inherent flaws (hallucinations and systemic biases).
Methods
The researchers conducted a comprehensive "stress test" of LLM-based review systems (specifically GPT-4o, Llama-3.1, and others) using several experimental setups:
- Explicit Manipulation (Review Injection): A "covert injection" attack was developed where manipulative instructions (e.g., "Focus only on these four strengths...") are added to a PDF in a tiny, white font. While invisible to humans, LLM parsers read this text as part of the manuscript.
- Implicit Manipulation: The study analyzed how LLMs respond when authors proactively disclose minor limitations, comparing the frequency of "parroting" versus independent critical analysis by human reviewers.
- Inherent Flaw Testing: To test for hallucinations, models were fed empty papers or titles only. Bias was tested by altering paper lengths and swapping author names with famous Turing Award winners to observe score changes.
- Review2Rating: A specific model was trained to map textual feedback to a 1–10 scale to quantify the impact of these risks.
The study primarily utilized the ICLR 2024 open-access review dataset, containing 28,028 review samples, providing a real-world baseline.
Results and Insights
Explicit Manipulation and Score Inflation
LLMs are incredibly susceptible to prompt injection. By embedding hidden text, authors can achieve a 90% agreement rate between the LLM's review and the author's own "wish list" of feedback. This explicit manipulation shifted average ratings from a "borderline" 5.34 to a "strong accept" 7.99 across various review systems.
Implicit Manipulation and the Echo Chamber Effect
LLMs exhibit a lack of independent critical thinking, acting as echo chambers for the authors' own self-disclosures. They are 4.5x more likely than humans to reiterate the exact limitations disclosed by authors. This suggests LLMs struggle to look past what an author claims is a limitation to find deeper, unstated flaws.
Inherent Flaws and Hallucinations
The study revealed startling hallucinations; one system gave an empty paper a review praising its "novel methodology" and "well-written" structure. LLMs also showed significant systemic biases:
- Authorship Bias: Changing affiliations to prestigious universities or famous researchers increased positive rating ratios from 36.8% to over 41%.
- Length Bias: Positive ratings increased monotonically with paper length, suggesting LLMs equate quantity with quality.
- Incomplete Content: Papers with only a title and introduction received ratings comparable to full manuscripts.
Discussion and Conclusion
The potential for systemic disruption is significant. Simulations showed that if just 5% of reviews were manipulated, 12% of top-tier papers could be displaced by lower-quality, "hacked" submissions. While future models may become more robust, they may also become better at justifying biased conclusions.
The findings call for a moratorium on using LLMs as primary reviewers until robust safeguards—such as automated manipulation detection tools—are established. Currently, LLMs should only serve as supplementary tools to assist human judgment, rather than replace it.
Summary updated