TL;DR
This paper quantifies the "homogenization" of peer review in the age of Large Language Models (LLMs), revealing a sharp rise in critique overlap across major AI conferences and journals. By analyzing claim-level data, the author demonstrates that reviews flagged as AI-generated are significantly more redundant than human ones, suggesting that the independent diversity of thought essential to peer review is rapidly eroding.
What Problem Does This Paper Address?
The fundamental value of peer review rests on the independence of reviewers. Ideally, multiple experts look at a paper from different angles, catching a diverse set of flaws. If Reviewer A misses a statistical error but Reviewer B catches it, the system works.
However, as LLMs become ubiquitous tools for drafting or "polishing" reviews, there is a growing concern that they are driving a "generative monoculture." If reviewers rely on AI that tends to converge on a narrow set of common, surface-level critiques (like "missing baselines" or "clarity issues"), the collective feedback becomes redundant. This paper seeks to measure whether reviews of the same paper are becoming more similar over time and whether this trend is directly linked to LLM usage.
Methods
The author moves beyond simple text-similarity metrics, which often conflate writing style with actual content. Instead, they employ a multi-stage pipeline:
- Claim Extraction: Using a high-reasoning model (GPT-6 Luna), the author extracts "atomic critique claims"—specific, actionable problems identified by a reviewer.
- Problem Grouping: A second pass groups these claims across different reviews of the same paper. If two reviewers complain about the same specific figure or the same missing experiment, it is counted as a "shared problem."
- Overlap Metric: The primary metric is the critique overlap coefficient (a version of the Simpson coefficient), defined as the number of shared problems divided by the size of the smaller critique set.
- LLM Detection: The study uses the MELD detector to flag reviews likely written or heavily edited by LLMs.
- Style Control: To ensure the results weren't just an artifact of LLMs having a similar "voice," the author conducted experiments where human reviews were rewritten by AI. They found that style changes only accounted for a small fraction of the observed overlap, confirming the homogenization is happening at the content level.
Datasets
The study utilizes a massive longitudinal dataset of nearly 500,000 reviews across:
- Major ML Conferences: ICLR (2020–27), NeurIPS (2021–25), and ICML (2025–26).
- Scientific Journals: Five sources including PLOS, F1000, and SciPost (2019–26).
- Validation Sets: Human-labeled critique sets from Nature-family and NeurIPS reviews were used to verify the accuracy of the automated claim extraction.
Key Insights and Ideas
- The Diminishing Returns of the "Third Reviewer": Historically, adding a third reviewer significantly expanded the pool of identified problems. The paper finds this "novelty" is shrinking; in 2027, a third ICLR review added roughly 10% fewer new problems than it did just a year prior.
- The "Flagged-Pair Gap": There is a statistically significant gap in redundancy. Pairs of reviews where both are flagged as AI-generated overlap much more than pairs of unflagged (presumably human) reviews.
- Shift in Critique Nature: The types of critiques are changing. While "clarity" and "typo" complaints have fallen, generic requests for more experiments and ablations have risen. This suggests LLMs may be defaulting to "safe," template-like criticisms that are easy to generate but offer less specialized insight.
- Homogenization Beyond the Detector: Interestingly, overlap is rising even among unflagged reviews. This suggests either that LLM use is more widespread than detectors can catch, or that human reviewers are subconsciously mimicking the homogenized style and focus of the AI-generated content they encounter.
Key Results
- Rising Redundancy: At ICLR, measured critique overlap jumped from 12.0% in 2026 to 21.3% in 2027. Similar year-on-year rises were observed at NeurIPS and ICML.
- The AI Redundancy Penalty: At ICLR (2026–27), pairs of LLM-flagged reviews had an overlap 4.5 points higher than unflagged pairs. Pooled across five journal sources, this gap rose to 5.4 points.
- Consistent Findings Across Venues: The "flagged-pair gap" was positive and significant across all four conference sets and most journal sources, indicating this is a systemic shift in scientific communication, not an isolated incident.
Limitations and Future Work
- LLM-in-the-Loop: The study uses an LLM to measure the impact of LLMs. While the author validated this with a "council" of different models, the possibility of shared systematic errors in how AI models perceive and group "claims" remains.
- Detector Limitations: AI detectors are not perfect. An "unflagged" review is not a guaranteed human review; it may simply be a more sophisticated or "stealthy" use of AI.
- Causality: While the correlation between AI flagging and high overlap is strong, the study cannot definitively prove that LLM use causes the rise in overlap among unflagged reviews.
- Future Directions: The author suggests investigating "trace-free" LLM use—where a human takes ideas from an AI but rewrites them entirely—and tracking whether these homogenization levels persist as peer review policies evolve.
Summary updated