Benchmark Errors as Natural Contamination Canaries

Every widely used benchmark ships with wrong answers. A careful re-annotation of MMLU estimates that about 6.5% of its questions are flawed, reaching 57% in Virology (Are We Done with MMLU?). A full audit of the TACRED relation-extraction dataset found 23.9% of labels incorrect (Re-TACRED). These errors are usually treated only as a nuisance that caps measurable accuracy. We propose to use them as evidence instead. A model that has never seen the answer key should rarely pick an answer that is factually wrong and specific to one benchmark's mistaken key. A model that memorized the key will pick it. The MMLU auditors saw this anecdotally: frontier models chose MMLU's erroneous "EU and USA army teams" answer to an Ebola question over medically sound options. We turn that anecdote into a calibrated statistical test for contamination, meaning the leakage of test questions and answers into training data.

The test has two properties current contamination detectors lack. First, it needs only the model's chosen answers, so it works on closed commercial APIs and can be applied retroactively to benchmarks that were already published. Second, it asks whether the model reproduces a specific falsehood, not whether the model's likelihood of a text is unusually high. The likelihood gap is exactly the signal that reinforcement learning erases in current detectors (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Whether memorized falsehoods survive that same post-training is the central empirical question of this proposal.

Core Claim

A benchmark's verified label errors form a set of free, unforgeable canaries: marker items whose reproduction reveals exposure to the answer key. Counting how often a model agrees with the erroneous key is not enough on its own. Some wrong keys encode common misconceptions that a clean model might also choose. The proposal's key ingredient is therefore a per-item null rate: how often the model would pick the wrong key through honest error rather than memory.

We estimate that rate for each flawed item $i$ as $q_i$, the probability that an uncontaminated model selects the erroneous key. We obtain $q_i$ from two sources. One is a panel of reference models trained on corpora decontaminated against the benchmark. The other is the audited model's own behavior on meaning-preserving paraphrases with shuffled options, which break surface recall but keep any misconception intact.

With $x_i \in \{0,1\}$ indicating that the audited model chose the erroneous key on the original item, the test statistic is

$$ Z = \frac{\sum_i (x_i - q_i)}{\sqrt{\sum_i q_i(1-q_i)}}, $$

compared against its null distribution. Under the null of no answer-key memorization, the $x_i$ are independent Bernoulli($q_i$) draws, so their sum follows a Poisson-binomial distribution. We compute exact p-values from that distribution and use $Z$ as a readable summary. The power of the test grows with the number of flawed items and with the gap between the memorized agreement rate and $q_i$. This makes high-error subjects and datasets the most informative, which inverts their usual role as the least useful parts of a benchmark.

Research Program

The work has three parts.

Building the canary inventory. For MMLU we restrict the Redux annotations (Are We Done with MMLU?) to the "wrong ground truth" category, where a different option is demonstrably correct. Phrasing ambiguities and unanswerable items are excluded, because a clean model choosing their key says little. For TACRED, the Re-TACRED corrections supply thousands of items where the original label disagrees with the corrected one. Many of these are spurious NO_RELATION keys (Re-TACRED). That class is a high-base-rate "default" answer, so its $q_i$ will be large. Measuring whether calibration handles it is part of the study.

Estimating the null. We compare the reference-panel estimate of $q_i$ with the paraphrase estimate and check that they agree on models known to be clean. Paraphrases are produced by an LLM and then checked by humans on the flawed items only. This keeps annotation cost bounded to a few hundred items rather than the whole benchmark. It also sidesteps the problem that LLM rewriting can itself inject label noise (Detecting Benchmark Contamination Through Watermarking).

Validating in controlled contamination. We inject the benchmark, with its true erroneous keys, into training at known duplication counts and stages. We then ask three questions. Does the test fire? Does it stay silent on clean models? Does it survive clean post-training that is known to blind likelihood-based detectors?

This extends the injection design of Data Contamination: From Memorization to Exploitation in one respect. That work injected only correct labels and separated memorization of the text from exploitation of it on the downstream task. Our injections carry the benchmark's real errors, so we can measure directly whether models memorize and exploit wrong keys as readily as right ones. That paper left this question open.

A byproduct is a diagnosis of a confound in existing detector evaluations. On a flawed item, a clean but capable model scores high loss against the benchmark key simply because it answers correctly. We report detector AUROC separately on clean items and flawed items to measure how much label noise distorts published detector numbers.

Baselines

Naive reference points. The do-nothing reference is uncorrected error agreement: the raw fraction of flawed items on which the model picks the erroneous key, thresholded without any null model. It shows what the calibration buys. The ablation that isolates our key ingredient replaces per-item $q_i$ with a single global null rate. That rate is the model's average probability of choosing any specific wrong option on clean items. If this global version performs as well as the per-item version, the per-item calibration is unnecessary and we will say so.

Likelihood-based detectors. The strongest post-hoc competitors are LiRA and loss-based membership tests as run in On The Fragility of Benchmark Contamination Detection in Reasoning Models. That paper reports LiRA at 89.1% AUROC after contamination by supervised fine-tuning (SFT), meaning training on benchmark question–answer pairs. It reports roughly a 14-point drop after one epoch of clean GRPO, a reinforcement-learning method, with reference-free methods near 50%. After contamination with chain-of-thought reasoning data, it reports 50–58% AUROC across all ten detectors. We run these detectors with identical injected models, identical item sets, and the same reference models that our null uses, so they get equal access to side information.

Sampling-based detector. We include CDD from Generalization or Memorization. CDD flags contamination when repeated samples collapse onto near-identical outputs, and the paper reports 21.8–30.2% gains over n-gram and LLM baselines on DETCON. It is the natural black-box competitor, so we give it the same sampling budget as our paraphrase-based null estimation.

Watermarking. Proactive watermarking (Detecting Benchmark Contamination Through Watermarking) reports $p<10^{-3}$ for a 5% contamination-driven gain on ARC-Easy with 1B models trained on DCLM, a large public pretraining corpus. It needs rewriting before release, so we compare to it only in the controlled setting, at matched contamination levels.

Criteria for beating the baselines. We count as success three outcomes. The test matches LiRA after SFT. It stays clearly above chance after clean GRPO, where LiRA degrades. It holds false-positive rates at nominal level on decontaminated models. No reported result was found on Lacuna for error-agreement detection. All numbers for it will be new.

Why not just…?

Why not just run Min-K% or LiRA, which already work after supervised contamination? They measure whether a text is unusually likely under the model. The fragility paper shows that clean reinforcement learning flattens exactly that likelihood gap. Our test asks a behavioral question: does the model still assert the benchmark's falsehood? Whether that survives RL is the key open question. RL rewarding correct answers could plausibly push models away from memorized wrong keys, which would weaken our test too. The Stage I and Stage II experiments settle this directly. If error agreement collapses alongside LiRA, we will report the negative result, and it would still show that RL removes memorized errors while leaving inflation on correct items intact.

Why not just watermark benchmarks before release? For new benchmarks, watermarking is a strong option. It cannot audit the benchmarks already embedded in every leaderboard and every pretraining crawl. Its LLM rewriting also risks corrupting meaning in technical domains, as its authors note. Label errors already exist in released benchmarks, cost nothing to plant, and cannot be stripped by a trainer who does not know which items are flawed. The two methods are complementary. We compare their statistical power at matched contamination to show how many natural canaries equal one watermark.

Why not just measure the accuracy drop on rephrased or fresh questions, as MMLU-CF and HumanEval_R do? This is the most serious alternative. MMLU-CF reports an average drop of about 14%, with o1 falling from 92.3% to 80.3%. But a drop on new questions confounds contamination with changes in difficulty and distribution, and it gives no per-model significance level. Our paraphrase-based null estimate is a within-item version of the same idea, applied only to flawed items. There, the memorized answer and the correct answer point in opposite directions, which makes the signal sharper. We include whole-benchmark paraphrase accuracy drop as an additional baseline. If it detects contamination as reliably and at the same false-positive rate, the canary framing adds little, and we will narrow our claims to the closed-API and retroactive settings.

Why not just clean the benchmark and move on? Cleaning is worth doing, but it destroys the canaries. Automated cleaning is also unreliable: Claude 3 Opus with retrieval reached 83.91% recall with low precision, and a Llama-3 8B detector trained on LabelChaos, a synthetic dataset of MMLU-style errors, reached an F2 score of 44.75 (Are We Done with MMLU?). We recommend reporting accuracy on clean items and error agreement on flawed items side by side, rather than deleting the flawed items.

Proposed Experiments

Controlled injection. We continue pretraining two open base models from the CANARY testbed, Qwen2.5-0.5B and Llama-3.2-1B, on web text into which MMLU, including its erroneous keys, is injected at 1, 4, 16, and 64 duplicates. Matched controls see MMLU-decontaminated text only. We then apply the two pipelines of the fragility paper. Stage I is SFT contamination followed by clean GRPO. Stage II is chain-of-thought SFT contamination, using MMLU-style questions in place of that paper's math benchmarks.

Relation-extraction replication. We repeat the design for TACRED in prompted relation extraction. The very different error structure there, dominated by NO_RELATION, stresses the null calibration.

Decisive checks. Five results decide the proposal. First, on clean controls, the false-positive rate must match the nominal level for both the per-item null and the global-null ablation. This decides whether calibration is needed. Second, after SFT, detection AUROC across duplication counts must be at least comparable to LiRA. Third, after clean GRPO and in Stage II, error agreement must remain above chance while likelihood detectors fall toward 50%. This is the main claim, and it can fail. Fourth, we directly measure whether wrong keys are memorized and exploited at the same rate as correct keys, answering the open question from Magar and Schwartz. Fifth, we split existing detectors' AUROC by clean versus flawed items to quantify how label noise distorts them.

Deployment on released models. Only after these checks pass do we apply the calibrated test to publicly released open and API models on MMLU. We will report p-values per model and per subject, making no claims about specific vendors beyond what the statistics support.