Language models are now routinely scored on writing peer reviews. Benchmarks such as ReviewBench, FIRSTPASS and NLPeer draw on reviews that sit on the public web, often next to arXiv preprints of the same papers. A model can therefore look like a sharp reviewer simply because it recalls, loosely or exactly, what a human reviewer wrote about that paper. The usual tools for catching this fail where it matters. Checks on token probabilities need the model's internals and are known to collapse after reinforcement-learning post-training. Date cutoffs do not stop paraphrased leakage. Prompting another model to judge rewards fluent-looking text.
We propose a black-box, per-paper test built on one observation. Public papers usually have several independent human reviews. A model that genuinely analyzes the manuscript should agree with each reviewer roughly as much as the manuscript itself makes inevitable. A model that recalls one review will match that reviewer's specific, idiosyncratic points far better than it matches the reviewer's siblings. We measure this through sequence alignment, meaning a matching of the model's critique points to a human's critique points that allows skips and reorderings at a cost. We make the matching differentiable so that its cost function can be learned. The learned costs weigh an agreement highly when neither the manuscript nor the other reviewers explain it.
Core Claim
Contamination of review-generation benchmarks can be detected from generated text alone, at the level of a single paper, with a contrast between reviewers. The score compares the generated review's alignment to each human review of the same paper, after removing agreement that the manuscript explains. The largest excess over the sibling reviews signals recall.
The detector has two parts. The first is a contrastive null that no existing detector uses: sibling reviews combined with manuscript grounding. The second is a learned alignment scorer. It adapts d$^2$p, which differentiates through dynamic-programming alignment and learns gap penalties end to end. We use it to learn the content-dependent skip costs that that paper names as an open frontier. For example, skipping a generic "clarity could improve" point should be cheap, while skipping a reviewer's unusual technical objection should be expensive.
We are explicit about which part carries the weight. If an order-free matching with the same learned costs does as well, then the contrastive null is the contribution and the ordering is not. Our ablations are built to settle this.
Why this gap is real
Three results motivate the work. Soft Contamination Means Benchmarks Test Shallow Generalization finds semantic duplicates for 78% of CodeForces problems in Olmo3 training data. It also shows that such duplicates inflate scores by roughly 15–20% even at tiny contamination rates. Its detector is a global embedding cosine plus LLM annotation, and the paper itself notes that this misses structural leaks. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that ten probability-based detectors fall to 50–58% AUROC once models are post-trained with reinforcement learning or contaminated with reasoning traces. PaCoST compares confidence on original versus rephrased items. It needs white-box access and works only at the level of a whole dataset.
On the review side, the benchmarks guard against leakage with blunt tools. ResearchTown relies on a post-October-2023 cutoff, and FIRSTPASS strips Discussion and Conclusion sections. Neither audits whether generated reviews echo real ones. Kahneman4Review shows that LLM judges reward form: length explains about 74% of the apparent AI-review advantage. It also concedes that without the paper, the fidelity of a review cannot be checked. Our detector conditions on the manuscript by design, so these works both motivate our approach and argue against using an LLM judge as the scorer.
Research Program
Segmenting and grounding. We first split each review into a sequence of critique points. We use claim tagging in the style of SubstanReview, whose RoBERTa tagger reaches only 55.61 F1. That weakness is why we also test plain sentence segmentation. We then ground each point in the manuscript. NLPeer provides links between review comments and paper paragraphs, and we use these links as weak supervision for a grounding model.
The idiosyncrasy weight. Each human point $h$ in review $H_j$ gets an idiosyncrasy weight $w(h)$, which is learned but seeded with two features. One is how weakly the manuscript supports $h$. The other is how rarely sibling reviewers of the same paper raise it.
The alignment score. For a generated review $G$ we compute a soft alignment score $A_\gamma(G, H_j)$ with a d$^2$p-style dynamic program over the two point sequences. Here $\gamma$ is the smoothing temperature from Soft-DTW. In this program, the cost of matching two points is their learned embedding distance. The cost of skipping $h$ scales with $w(h)$.
The per-paper statistic. The statistic for paper $p$ with $k$ reviews is
$$S_p = \max_j A_\gamma(G, H_j) - \frac{1}{k-1}\sum_{j' \neq j^*} A_\gamma(G, H_{j'}),$$
where $j^*$ is the reviewer attaining the maximum. We calibrate it against a null distribution. The null is built from generations by a model whose training provably excludes the paper's reviews, and from permutations of reviewer identities.
Training. Because the dynamic program is differentiable, we train the point encoder, $w$, and the skip costs end to end. The objective is a ranking loss that separates contaminated from clean (model, paper) pairs in a controlled-contamination suite.
Secondary output. The grounding alignment also yields a review-to-manuscript fidelity score. We evaluate it against NLPeer's links, which targets the verification gap Kahneman4Review leaves open. We treat this as a byproduct, not the headline.
Baselines
Naive reference points. Two reference points anchor the comparison. The first is chance, 50% AUROC. The second is word n-gram overlap between the generated review and the best-matching human review, which is what standard decontamination does. We expect n-gram overlap to catch verbatim recall and to fail on paraphrase-contaminated models.
Unordered embedding similarity. The strongest text-only competitor is the method of Soft Contamination Means Benchmarks Test Shallow Generalization: cosine similarity between whole-review embeddings. We give it the same sibling-reviewer contrast, so that it differs from our method only in having no alignment. It is the strongest competitor because it is the published method for catching paraphrased leakage.
Probability-based detectors. These are the strongest method available when the model's internals are accessible. We run Min-K%, loss scoring and LiRA from On The Fragility of Benchmark Contamination Detection in Reasoning Models, and PaCoST, on the open models, using the human review text as the probe.
The Fragility paper gives reference figures for this class of detector on math and science benchmarks, not reviews. LiRA reaches 89.1% AUROC on models contaminated by supervised fine-tuning before reinforcement learning. It drops by about 14 points after one epoch of GRPO, a reinforcement-learning post-training method. Detectors average 50–58% AUROC on contaminated reasoning models. There is no reported result on Lacuna for detecting contamination in peer-review generation.
Beating this class means two things. On the fine-tuned-only models, our method must reach at least LiRA's AUROC. On the models that also received GRPO, it must hold its AUROC with a drop well under the 14 points LiRA loses.
LLM judge. We give a strong model the paper, the generated review and all human reviews, and ask which human review, if any, the generated review appears derived from. We expect the form bias documented by Kahneman4Review to show up here.
Ablations of the key ingredients. The ablation that isolates our key ingredients comes in three versions. The first removes ordering by replacing the dynamic program with an order-free Sinkhorn matching that keeps the learned costs. The second removes learned content-dependent skip costs, using scalar gap penalties as in d$^2$p. The third removes manuscript conditioning by setting $w$ to be uniform.
Fairness. All learned baselines share our training split, point encoder backbone and tuning budget. All are scored on the same held-out models and papers.
Why not just…?
Why not just use papers published after the training cutoff? Cutoffs are necessary but not sufficient. Preprints and reviews of closely related work predate the cutoff, and the soft-contamination paper shows that semantic neighbors inflate scores without exact overlap. Cutoffs also cannot audit the large pre-cutoff benchmarks already in use, such as NLPeer and the 2021–2023 ICLR portion of Kahneman4Review. We use post-2025 reviews as our clean null rather than as a remedy.
Why not just compare whole-review embeddings against sibling reviews? This is the real threat. The sibling contrast may do most of the work, and our unordered-embedding baseline is exactly this method. If it matches our detector on paraphrase contamination, we will report that ordering and learned alignment add nothing. Our bet is that points reproduced in the same sequence, and unusual points reproduced at all, survive paraphrasing better than a global vector does. The Sinkhorn and uniform-$w$ ablations separate these two effects.
Why not just use perplexity or PaCoST on the human review? Such methods need log-probabilities, which rules out commercial systems. The Fragility paper shows they degrade toward chance after reinforcement-learning post-training. We run them anyway, so that the benefit of losing white-box access is measured rather than asserted.
Why not just ask a frontier model whether the review was copied? Kahneman4Review finds that LLM judges collapse nine analytic dimensions into one surface factor and track length. Our judge baseline will show whether this also blinds them to paraphrased recall.
Proposed Experiments
Contamination suite. We build a controlled suite from NLPeer's ARR-22 and ACL-17 papers, which have multiple reviews each, and from ICLR/NeurIPS 2024 tasks in ReviewBench. We split papers into seen and unseen sets. We then fine-tune Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Llama-8B, the models used in the Fragility study, on (paper, one designated review) pairs. The pairs come in three forms: verbatim reviews, LLM-paraphrased reviews, and a low-dose condition that mimics the soft-contamination paper's regime of 4 in 10,000 training examples. We then optionally apply GRPO with a review-quality reward to test cloaking.
Primary metric. The primary metric is per-paper AUROC for contaminated versus clean papers. The secondary metric is accuracy at identifying which reviewer was leaked.
Generalization tests. We train on one model family and one contamination form, and test on the other. This is the decisive check against the detector simply learning fine-tuning artifacts.
Multi-round extension. FIRSTPASS pairs each round-one review with the author rebuttal and the round-two review. We test whether a model that leaked only the rebuttal thread reproduces reviewer concerns in the order they were resolved.
Real-world audit. We generate reviews with open and API models for 2021–2023 ICLR papers and for 2025 papers from Kahneman4Review. A valid detector should flag excess sibling-specific agreement only on papers the model could have seen.
Fidelity output. We report the fidelity score against NLPeer's links, alongside the MRR of 0.41 that NLPeer reports for guided skimming.
Failure criterion. The idea fails if order-free matching with the sibling contrast equals the full model under paraphrase and GRPO conditions. In that case the contribution narrows to the contrastive null, and we will say so.
References
- d$^2$p: Structured Soft Attention Is All You Need
- Soft Contamination Means Benchmarks Test Shallow Generalization
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- ResearchTown: Simulator of Human Research Community
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes
- Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
- Automatic Analysis of Substantiation in Scientific Peer Reviews
- NLPeer: A Unified Resource for the Computational Study of Peer Review
- Soft-DTW: a Differentiable Loss Function for Time-Series