Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer Review

Jizhou GuoarXiv 2026 · arXiv

TL;DR

As AI models are increasingly used to assist in academic peer review, this paper reveals a critical flaw: version drift. Newer versions of frontier models (like Claude and Gemini) are becoming significantly harsher, with "acceptance" rates for the same set of papers plummeting by over 50% in just a few months, even though their relative ranking of papers remains somewhat stable.

What Problem Does This Paper Address?

In the world of academic publishing, researchers and conference organizers are starting to use Large Language Models (LLMs) to provide feedback or even "anchor" scores for submissions. The underlying assumption is that an LLM score is a stable measurement of a paper’s quality.

However, AI providers (OpenAI, Google, Anthropic) update their models every few months. This paper investigates whether these updates change the "verdict" for the same paper. If a model version from February accepts a paper but the September version rejects it, the LLM isn't acting as a thermometer measuring quality—it's a moving target. Additionally, the paper tackles the "contamination" problem: can models accurately review papers they might have already seen in their training data?

Methods

The author conducted an audit of twenty frontier models released between February and November 2026. The study utilized a "prospective" design, meaning the analysis plans were registered and timestamped before the official ICLR 2027 decisions were released, ensuring the results weren't cherry-picked to match human outcomes.

Key experimental techniques included:

  • Version Ladders: Comparing successive versions of the same model family (e.g., Claude Opus 4.6 through 5.5) on identical prompts and papers.
  • Cross-Scoring: A clever "swap" experiment where the critique written by an older model was scored by a newer model (and vice versa) to see if the drift was in the writing of the review or the assignment of the final score.
  • Neutral Rewriting: Using a separate model to strip evaluative adjectives (like "significant" or "weak") from reviews to see if the underlying technical points alone would trigger a different score.
  • Memory Probes: Testing if models could identify authors or venues from paper titles alone to detect data leakage.

Datasets

The study focused on several batches of papers:

  • ICLR 2027 Submissions: A stratified sample of 24 papers (low, middle, and high human ratings) and larger random batches of up to 183 submissions.
  • ICLR 2026 Decided Papers: Used as a calibration set and for memory/contamination testing.
  • Award Papers: 24 papers that won "Best Paper" or "Outstanding Paper" awards at top-tier venues (NeurIPS, ICML, etc.) to see if models recognize "exceptional" work.

Key Insights and Ideas

  • The "Harshening" Trend: Across the board, newer models are becoming much more critical. For example, Claude Opus 4.6 put 82% of a sample on the "accept" side; by version 5.5, that share dropped to just 12%.
  • Ranking is Stable, Scale is Not: Models generally agree on which papers are better than others (ranking correlation remains high at ~0.73). However, they disagree wildly on where the "pass/fail" line should be.
  • Critiques are the Culprit: The drift isn't just a change in the final number. Later models actually identify more weaknesses. Even when those weaknesses are rewritten into neutral language, the later models still use them to justify lower scores.
  • The "Middle-Point" Collapse: LLMs are incredibly hesitant to give extreme scores. In 475 reviews, models gave a "1" (clear reject) or "4" (clear accept) only twice. Humans, by contrast, used the full scale much more frequently.
  • Memory Contamination is Real: Claude Opus 5.5 could name the first author of 27 out of 60 decided papers from the title alone with 100% accuracy. This suggests that using LLMs to "re-evaluate" old papers is essentially asking the model to "remember" the previous verdict rather than reason through the text.

Key Results

  • Massive Accept Share Drops: Between earlier and later versions of Claude Opus and Gemini, the share of papers "accepted" fell by 52 and 53 percentage points respectively.
  • Human Correlation: Interestingly, the self-estimated percentiles of three models correlated with the mean of two human reviewers more strongly than a third human reviewer did. This suggests LLMs are "human-like" in their rankings, even if their absolute scales are drifted.
  • Soundness Bias: For GPT models, a "2" (weak rejection) is the default for almost everything. They gave a soundness score of "1" (significant concerns) to nearly a quarter of all papers, but never gave the top score of "3."

Limitations and Future Work

  • Sample Size: While the study covers hundreds of reviews, some specific version-pair comparisons were limited to 24-60 papers.
  • Reasoning Effort: The models were tested at specific "reasoning effort" settings (e.g., OpenAI "medium"). It is unclear if increasing the compute/reasoning time would mitigate or exacerbate the drift.
  • Human Noise: The study compares LLMs to human reviews, which are themselves notoriously noisy and inconsistent.
  • Future Work: The author notes that the final validation against ICLR 2027 decisions will be reported after December 16, 2026. This will provide the ultimate "ground truth" to see which model version actually aligns best with the final human committee decisions.

Conclusion for Practitioners: If you are using LLMs for peer review, never use a fixed score threshold. A "3.0" from a model today does not mean the same thing as a "3.0" from the same model family six months ago. Any automated screening must be re-validated and re-anchored for every single model update.

Summary updated

Abstract

Venues and benchmarks read LLM review scores as measurements of a paper, while providers replace their models within months. We audit twenty frontier models from OpenAI, Google and Anthropic on ICLR 2027 submissions while they were under review, with tests registered before the decisions. Between an earlier and a later version of Claude Opus and of Gemini, the share of papers placed on the accept side falls by 52 and 53 percentage points on 183 and 60 randomly drawn submissions, and by 43 to 79 points in six further paired comparisons, with no paper moving up in any of them. Stating a 30% acceptance rate in the prompt leaves a gap of 79 points between the two Opus versions. When score-stripped critiques are swapped between versions, the later critiques alone lower the accept share by 35 points for Opus and 51 for Gemini; for Opus the drop is 35 points under a third-party scorer and 24 after a rewrite that keeps each technical point and removes evaluative wording. On 57 papers, the percentiles of Opus 4.6 and 5.5 correlate at 0.73 while their accept shares differ by 44 points. On 99 to 119 submissions, the self-estimated percentiles of three models correlate with two human reviewers more strongly than a third reviewer does (point estimates 0.17 to 0.21, 95% intervals between -0.04 and +0.42). All twenty models give a 2 or a 3 in 473 of their 475 reviews, against 60 of 72 human reviews. Retrospective tests on decided papers can draw on memory. Claude Opus 5.5 names the true first author of 27 decided ICLR 2026 papers from their titles alone, with no wrong name, and Claude Opus 4.6 accepted all 11 decided papers it said it recognized, against 5 expected. Thresholds on LLM-reviewer scores therefore have to be validated against human decisions for each model version; our tests against the ICLR 2027 decisions follow on 16 December 2026.
BibTeX
@misc{guo2026same,
  author = {Jizhou Guo},
  title = {{Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer Review}},
  year = {2026},
  eprint = {2611.20608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20608},
}
Figure