Many benchmarks quietly assume that each question has exactly the answers the annotators wrote down. In practice, a large share of valid answers is simply missing. Code retrieval benchmarks omit between 44% and 87% of relevant snippets (Hidden Positives). The DocRED relation-extraction benchmark misses roughly 66% of true relations (Huang et al.). This proposal turns that defect into a detector for a different problem: benchmark contamination, meaning test items that leaked into a model's training data. The reasoning is simple. When several answers are equally correct, a model that has never seen the benchmark has no reason to prefer the particular one the annotators happened to record. A model that memorized the benchmark does. So among a model's correct outputs, the share that coincide with the annotated answer, compared with the share produced by trusted clean models, is a behavioral fingerprint of contamination. It needs no access to model probabilities, no training corpus, and no rephrased copy of the benchmark.
Put in two sentences: missing labels mean benchmarks record an arbitrary subset of the correct answers, and only a model that has seen the benchmark knows which subset that is. We propose to measure that "canonical preference" as a calibrated statistical test, validate it under controlled contamination, and check whether it survives the reinforcement-learning post-training that is known to erase existing contamination signals.
Core Claim
Let $V_i$ be the set of valid answers for benchmark item $i$, established by exhaustive human re-annotation or an audited judge. Let $c_i \in V_i$ be the single answer the original benchmark annotated. For a model $M$, the canonical share on item $i$ is
$$s_i(M) = \Pr_M\big(\text{output} = c_i \mid \text{output} \in V_i\big).$$
For retrieval, $s_i(M)$ is estimated as the indicator that $c_i$ ranks above every other judged-valid candidate in $M$'s top ten. For set-valued tasks such as relation extraction, it is estimated as the model's recall on originally annotated relations divided by its recall on the relations only exhaustive re-annotation found.
Annotated answers are not chosen uniformly at random. They tend to be popular or early in the document, and Huang et al. show that 70% of added DocRED instances sit in the first three sentences. Clean models may therefore also prefer $c_i$. The test is thus relative to a pool of reference models $\mathcal{R}$ trained before the benchmark existed or with verified clean data. The contamination statistic is
$$\Delta(M) = \frac{1}{n}\sum_{i=1}^{n}\Big(s_i(M) - \bar s_i(\mathcal{R})\Big),$$
where $n$ is the number of items with $|V_i|\ge 2$ and $\bar s_i(\mathcal{R})$ is the reference pool's mean canonical share on item $i$. Significance comes from a paired test across items. The claim is that $\Delta$ is near zero for clean models and positive for contaminated ones. Unlike raw accuracy, $\Delta$ conditions on correctness, so it separates "knows the answer" from "knows which answer the benchmark wants."
Relation to Prior Work
The motivating evidence comes from missing-label work that operates at evaluation time. Hidden Positives re-annotates the top ten outputs of code retrievers with Claude-Opus-4.6, which agrees with human experts 87.3% of the time. It shows that benchmarks underestimate models that retrieve valid alternatives. That paper corrects scores; it never asks whether the uncorrected score rewards memorization of the canonical snippet. That is our question. Huang et al. supply the ideal test bed: a 96-document subset of DocRED re-annotated from scratch, with 3,308 relations against the original 1,214. They report that the ATLOP relation extractor drops from 62.5 to 44.5 F1 on the exhaustive set. That gap is the kind of annotation-shaped signal a contaminated model should exaggerate.
The contamination detectors we compare against make a shared assumption. PaCoST compares a model's self-reported confidence on original versus rephrased questions with a paired t-test, which requires token probabilities. ConStat compares a model's score on the original benchmark with its score on synthetic rephrased benchmarks, using a difficulty curve fit over reference models. Both treat the reference answer as complete. As a result, a clean model that finds unannotated positives looks artificially weak on the original benchmark, and this can bias ConStat's inflation estimate. We borrow ConStat's idea of calibrating against a reference-model pool but apply it to preference within the valid set rather than to accuracy. That avoids generating rephrased benchmarks, which may themselves omit long-tail answers.
The main constraint comes from On The Fragility of Benchmark Contamination Detection in Reasoning Models. It shows that GRPO post-training shrinks the log-probability gap between seen and unseen items. The LiRA detector falls from 89.1% AUROC on SFT-contaminated models, and after contamination with chain-of-thought data, ten detectors average only 50–58% AUROC. Inflated pass rates survive. Our hypothesis is that reinforcement learning flattens confidence but not the discrete choice of which valid answer to emit, because the reward is paid on the canonical answer. This is a hypothesis, not a result, and testing it is the decisive experiment.
Test of Time shows that rephrasing launders contamination: influence-function attribution falls from 77.5% to 17.5% hit rate once questions are transformed. This constrains the design. It warns that a memorizing model might emit a paraphrase of $c_i$ rather than $c_i$ itself, so "output = $c_i$" must be judged by semantic or functional identity, not string match.
Research Program
The work has three parts.
The first part builds multi-answer evaluation sets where $V_i$ is trustworthy. For DocRED we use the human-exhaustive $D_{Scratch}$ labels, so no language-model judge is involved. For code retrieval (CoSQA, CoSQA+, CSN-Adv, RepoQA) we reuse the judge protocol of Hidden Positives. We add a human-audited subsample and a second judge from a different model family, so that disagreement between the two judges bounds judge error.
The second part is a controlled contamination study where ground truth is known by construction. We take multi-answer items and, crucially, randomly re-designate which valid answer is canonical, which creates a fresh benchmark variant nobody has trained on. We then fine-tune open models on the canonical versions of half the items at several exposure levels. The models are Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Llama-8B, the setting of the fragility paper, plus a small code embedder for retrieval. Random designation removes the popularity confound entirely, since the null expectation is exactly the clean model's preference. We then apply one to three epochs of clean GRPO and measure whether $\Delta$ survives.
The third part applies the calibrated test to public model families on the original, non-randomized benchmarks. There the reference pool must absorb the natural popularity of canonical answers.
A secondary analysis addresses the inverse question that Hidden Positives leaves open: how much of a model's lead on the original metric disappears when scored against $V_i$, and whether that disappearing share correlates with $\Delta$.
Baselines
The naive reference point is to do nothing: report original-benchmark scores and assume no contamination. In the controlled study this corresponds to an AUROC of 50% for flagging contaminated items.
The strongest black-box competitor is ConStat, which reports detecting 98.4% of sample-specific contamination in controlled experiments against 71% for the best prior baseline. It also reports a gap of about 6.6% for Llama-3-70b on ARC. We run it with the same reference pool and the same compute budget for generating rephrased references. Beating it means matching its detection rate on SFT contamination while retaining clearly higher detection after GRPO, where we expect rephrase-based accuracy gaps to persist but be confounded by missing labels.
The strongest white-box competitors are LiRA and the loss detector as run in the fragility paper. Their reported numbers are 89.1% AUROC for LiRA under SFT contamination, dropping about 14% after one epoch of GRPO. We rerun them on our own contaminated checkpoints, not only quoting their numbers. Success is an AUROC for $\Delta$ that stays well above their post-GRPO values on identical checkpoints. PaCoST is run as a second white-box reference. It reports significance flags rather than AUROC, so no directly comparable number was found on Lacuna. An exact n-gram overlap check against the fine-tuning data is included as the cheapest sanity baseline.
Three ablations isolate the key ingredient. The first drops the conditioning on validity and uses raw accuracy on $c_i$ against the reference pool, which is essentially an accuracy-gap detector; this tests whether conditioning on $V_i$ matters. The second uses annotator-chosen instead of randomly designated canonical answers in the controlled study, which measures how much popularity bias the reference pool must absorb. The third swaps human-exhaustive $V_i$ (DocRED) for judge-derived $V_i$, which measures sensitivity to judge error.
Why not just…?
Why not just re-annotate with an LLM judge and report corrected scores, as Hidden Positives does? This is the closest threat, because the gap between original and corrected scores already carries our signal in unnormalized form. Corrected scores alone cannot flag contamination: a model can score well on both metrics by being good. Only the within-valid-set preference relative to clean references separates skill from memorization. The first ablation tests exactly this. If raw corrected-minus-original gaps detect contamination as well as $\Delta$, our contribution shrinks to a reporting convention, and we will say so.
Why not just use rephrased benchmarks, as ConStat and PaCoST do? Rephrasing generates new questions but keeps the old incomplete answer keys. Rephrasing is also exactly what Test of Time shows to be a laundering channel. Our test instead holds the question fixed and varies which valid answer is credited, a separate axis. The head-to-head on shared checkpoints settles which axis survives reinforcement learning.
Why not just check performance drops after the training cutoff? Test of Time finds no systematic decay on generated questions (+2.19 points on post-cutoff physics), even when influence functions show the source document drives the output. Temporal splits also cannot be applied to fixed legacy benchmarks like DocRED, which are exactly where missing labels are best documented.
Why not just use n-gram overlap? It requires the training corpus, which closed models do not expose. PaCoST and ConStat both report that it misses paraphrased or answer-only leaks. We keep it only as a sanity baseline in the controlled setting, where the corpus is known.
Proposed Experiments and Decisive Checks
The first decisive check is the randomized-canonical controlled study. On clean checkpoints, $\Delta$ must be statistically indistinguishable from zero; this is the false-positive control. On SFT-contaminated checkpoints, $\Delta$ must be significantly positive, with item-level AUROC reported against contaminated/clean labels. After GRPO, the idea lives or dies on whether $\Delta$ retains its AUROC while LiRA and loss detectors collapse toward the 50–58% range reported by the fragility paper. If $\Delta$ collapses too, we report that discrete answer choice is also flattened by reinforcement learning. That is itself a useful negative result, because it would mean current post-training hides contamination at the behavioral level, not only in confidence.
The second check uses DocRED with human-exhaustive labels. We compare models' recall on the 1,214 originally annotated relations against their recall on relations found only in $D_{Scratch}$, relative to reference extractors trained without Wikipedia-derived exposure. A contaminated model should show a markedly steeper ratio than ATLOP's pattern.
The third check covers the code retrieval benchmarks with dual-judge validity sets. We report $\Delta$ across public embedders along with its stability when the second judge replaces the first. A judge that itself memorized canonical answers would shrink $V_i$ toward $c_i$ for all models alike. Because $\Delta$ is a paired difference against references, such a shift should largely cancel, and the judge-swap test checks that.
Throughout, the metrics are item-level AUROC in controlled settings, benchmark-level detection at a fixed 5% false-positive rate on clean references, and the estimated share of original-metric inflation that disappears under exhaustive scoring.
References
- Hidden Positives: Why Code Retrieval Benchmarks Underestimate Model Quality
- Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination