When a language model scores well on a public benchmark, we usually cannot tell whether it learned the skill or saw the test questions during training. Current contamination detectors can often answer a coarse version of the question: "was some of this dataset in training?" They are much weaker on the question that matters for fixing a leaderboard number: "which items leaked, and what would the score be without them?" This proposal treats that second question as a multiple-instance learning problem. Multiple-instance learning trains a model from labels attached to groups of examples ("bags") rather than to individual examples. The idea comes from relation extraction, where a label such as "Jobs founded Apple" is attached to every sentence that mentions both names, even though only a few of those sentences actually express it. We learn to point at individual leaked benchmark items using only dataset-level labels of the form "this model's training data contained part of this benchmark." The benchmark plays the role of the bag and its items play the role of the sentences.
The fit is closer than an analogy. Real contamination is partial and sparse: a few percent of the test items are leaked, and the clean majority drowns out their signal. That is the regime where the relation-extraction literature found that averaging fails and selective attention succeeds. Item-level ground truth is also almost never available for real models, while dataset-level evidence often is: known pretraining corpora, confirmed leaks, and controlled fine-tunes. A detector that learns from bag labels can therefore be trained on the evidence that actually exists. Its output is concrete: a ranked list of suspected leaked items, and a benchmark score recomputed without them.
Core Claim
Contamination can be localized to items by a two-level attention model. The upper level weights the items within a benchmark. The lower level weights the variants within an item, such as rephrasings, metadata-guided completions, and in-context probes. We claim that this model, trained only on bag-level contamination labels, can reach item-level localization close to a detector trained with item labels. We also claim that it keeps working after reinforcement-learning post-training hides the log-probability traces most detectors rely on. Two design choices carry the claim. The first is that the attention query depends on each item's estimated difficulty, so an item is flagged when the model does better on it than its difficulty predicts, not merely because it looks unusual. The second is that the per-item evidence mixes likelihood-based signals with performance-based signals, meaning whether the model answers the item and its rephrasings correctly. Performance inflation is the part of contamination that post-training does not erase.
Why this is open, and what it builds on
The contamination side supplies the evidence that motivates the gap. Detecting Data Contamination in LLMs via In-Context Learning (CoDeC) shows that memorized examples lose confidence when unrelated in-context examples are prepended, while unseen examples gain confidence. It reaches a dataset-level AUC of 99.9%, but it averages this effect uniformly over items. Its authors name sample-level inference and an ambiguous 60–80% score zone as open problems, and partial contamination is exactly what produces that ambiguous zone. Time Travel in LLMs probes items with prompts that name the dataset and split and then asks the model to complete a prefix, a probe it calls Guided Instruction. It aggregates results with significance tests over about ten sampled instances, with no model of mixed bags. ConStat redefines contamination as performance that does not generalize to rephrased or sister benchmarks, measured against a curve fit over many reference models. It works at the level of whole benchmarks. We borrow its performance-based view and push it down to items.
Two papers constrain the design. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that a single epoch of GRPO, a reinforcement-learning method, on clean data collapses the loss gap between seen and unseen items. After it, LiRA drops from 89.1% AUROC and ten detectors average 50–58% on a distilled reasoning model. Yet in the same setting the pass@1 inflation from contamination persists, at 8.8% and 11.7%. This is why performance features must be part of each item's evidence. Confounders in Instance Variation for the Analysis of Data Contamination shows that harder rephrasings lower scores for reasons unrelated to memorization. It also shows that diverse phrasings dilute the memorization boost, from 71.0% to 61.4%, and that averaging over variants therefore confounds difficulty with leakage. That result motivates attention over variants conditioned on difficulty, rather than a plain mean. The paper does not group variants into bags or learn weights over them. Finally, A Narrowing Geometry in Contaminated Reasoning finds that leaked items are processed through a low-dimensional activation subspace. For open-weight models this gives an optional white-box item feature.
The relation-extraction side supplies the method. Improving Distantly Supervised Relation Extraction with Neural Noise Converter and Conditional Optimal Selector shows two things. Averaging hides a single informative sentence among many "no relation" distractors. A conditional selector that lets any strong positive instance decide the bag fixes this. Incorporating Relation Paths in Neural Relation Extraction documents the opposite failure: picking only the top instance throws away complementary evidence. Multi-level Attention-Based Neural Networks for Distant Supervised Relation Extraction contributes the two-tier word and sentence attention we mirror as variant and item attention. It reports P@100 of 84.0% against 76.0% for single-level attention on NYT10. Neural Knowledge Acquisition via Mutual Attention Between Knowledge Graph and Text uses external knowledge as the attention query. Our difficulty-conditioned query plays the same role: difficulty is the side information that tells the model what "unexpected success" means for this item. None of this work touches contamination. None of the contamination work uses bag-level learning.
Research Program
Each item $i$ of a benchmark $D$, evaluated against a target model $M$, is represented by a small bag of variants. These are the original item, $K$ automatically generated rephrasings, a guided-completion probe, and CoDeC-style in-context probes. Each variant yields a feature vector: the correctness of the answer, the loss and the change in loss under context, and the completion overlap under guided versus general prompting. Separately, we estimate each item's difficulty $d_i$ from the average accuracy of a pool of clean reference models on the item and its rephrasings. This is an item-level analogue of ConStat's hardness correction and needs far fewer models than a full curve fit. A variant-level attention layer pools the variant features into an item embedding $h_i$. An item-level layer then produces the bag score:
$$ s(D, M) = \max\Big(\sum_i a_i\, g(h_i),\ \max_i g(h_i)\Big), \qquad a_i = \frac{\exp\big(q(d_i)^\top h_i\big)}{\sum_j \exp\big(q(d_j)^\top h_j\big)}. $$
Here $g$ maps an item embedding to a contamination logit, and $q(d_i)$ is a learned query computed from the item's difficulty. The outer maximum is the conditional-selector idea: one sufficiently strong item can make the bag positive without being averaged away. Training uses only bag labels, with binary cross-entropy on $s$. At test time the per-item logits $g(h_i)$ and the weights $a_i$ serve as item-level leak scores. The corrected score removes items above a threshold calibrated on held-out bags and recomputes accuracy.
Training bags come from two sources. Synthetic bags are made by fine-tuning open models, including Llama-2 7B, Mistral-7B and Qwen2.5-7B-Instruct, all of which appear in the cited setups. Each fine-tune leaks a random fraction of a benchmark (1%, 5%, 20%, 50%) in one of three forms: verbatim, rephrased, or answer-only. Half of these models then receive a clean GRPO epoch, following the fragility paper's protocol. A matched model fine-tuned on the same data without the leak gives the true clean accuracy. Real bags come from corpora with documented membership, namely The Pile, Dolma and Nemotron-CC as used by CoDeC, together with benchmark partitions for which dataset-level leaks have been reported. The synthetic bags also have item labels, but the MIL model never sees them. We use them only for evaluation and for the supervised upper-bound baseline.
Baselines
The naive reference point is the per-item CoDeC signal, thresholded, with the leaked fraction estimated as the share of items that cross the threshold. This is exactly how CoDeC turns item signals into a dataset score. It is what a practitioner would do today, and the ambiguous-zone problem is inherited directly from it. CoDeC reports 99.9% dataset-level AUC on pretraining corpora, but on Lacuna we found no reported result for item-level CoDeC localization under partial contamination. We run it on identical bags with the same probe budget per item.
The strongest existing detector before reinforcement learning is LiRA, a likelihood-ratio test against reference models. It reaches 89.1% AUROC in On The Fragility of Benchmark Contamination Detection in Reasoning Models and falls to 50–58% after GRPO on distilled reasoning models. We give it the same reference-model pool used for our difficulty estimates. Beating it means matching it within noise before GRPO and staying well above 58% AUROC afterward. At the benchmark level, ConStat is the strongest performance-based competitor. It reports $\delta \approx 8.2\%$ for Mistral-7B on GSM8K and $\delta \approx 6.6\%$ for Llama-3-70B on ARC. We compare its $\delta$ against our contamination-corrected score gap on the synthetic models, where the true gap is known. Winning means lower absolute error in the estimated inflation with fewer reference models. Guided Instruction from Time Travel, with its 92–100% partition-level accuracy on closed models, is included as a black-box item feature and as a standalone baseline on the closed-model bags.
The upper bound is a supervised item classifier trained with true item labels on the same synthetic models and the same features. The ablations isolate each ingredient in turn. The first replaces the difficulty-conditioned query with a fixed query. The second removes the performance features and keeps only likelihood features, which should fail after GRPO. The third replaces item attention with a plain mean. The fourth replaces it with a hard maximum. The fifth collapses the variant level so that only the original item is scored.
Why not just…?
Why not just average each item's scores over its rephrasings and threshold the result? Averaging is what Confounders in Instance Variation shows to be confounded: hard templates drag scores down whether or not the item leaked, and diverse phrasings dilute the boost from 71.0% to 61.4%. The mean-pooling ablation tests this directly. We expect it to fail specifically on leaks that are only rephrased and on hard items. If it does not fail there, the variant attention is unnecessary and we will say so.
Why not just take the maximum signal over variants and items? This is a real threat, because leaks are sparse and a maximum is well suited to sparse positives. The relation-extraction evidence (Incorporating Relation Paths) says that a maximum discards complementary evidence. Here the analogous risk is that an unusually easy clean item produces a spurious peak. Our score already contains a maximum term, so the ablation that keeps only the maximum asks whether the attention term adds anything. We expect the gap to be largest at 20–50% leak fractions and after GRPO, when individual signals are weak but many items each carry a little evidence.
Why not just train a supervised item-level classifier on simulated contamination, where item labels are free? This is the most serious alternative, and it is our upper-bound baseline rather than a straw man. Its weakness is transfer. Simulated leaks are one fine-tune with a chosen format, while real leaks come from repeated, paraphrased web copies inside pretraining. Only the bag-level route can learn from the real bags. The decisive comparison has two halves. On synthetic held-out models, MIL trained without item labels should come within a few points of item-level AUPRC of the supervised model. On real Pile and Dolma bags, where a verbatim-overlap subset gives partial item ground truth, the MIL model trained with real bags should localize better than the supervised model trained only on simulated data. If the supervised simulator wins on real bags too, the case for MIL reduces to a convenience argument, and we would report that.
Why not just move to procedural benchmarks like A²RBench? New benchmarks do not audit the MMLU, GSM8K and ARC numbers that existing models are still compared on. Correcting those scores is the use case this proposal targets.
Proposed Experiments
The main testbeds are GSM8K, ARC-Challenge and MMLU, all used by ConStat and CoDeC, and the reasoning suite of AIME, AMC, GPQA Diamond, OlympiadBench and Minerva Math from the fragility study, which supplies the post-GRPO setting. The CONDA multi-digit addition set (Confounders in Instance Variation) serves as a controlled check on difficulty: digit count gives exact difficulty, and seven templates give graded rephrasing hardness. Training and test bags are split by base model and by benchmark, so the detector is never evaluated on a model–benchmark pair it was trained on. The metrics are bag-level AUROC, item-level AUPRC for leaked items, and absolute error of the corrected accuracy against the matched clean twin. Each is reported across leak fraction, leak format, and before versus after GRPO.
Three results would decide the proposal. The first is item-level AUPRC after clean GRPO. Likelihood-only detectors should sit near chance here, while the full model with performance features should stay clearly above it. The second concerns the ambiguous zone: on bags whose CoDeC score falls between 60% and 80%, the model should separate partially contaminated bags from clean bags drawn from a similar distribution. The third is estimation of the corrected score. On held-out synthetic models, removing flagged items should recover the clean twin's accuracy more accurately than ConStat's $\delta$, while using fewer reference models. If the difficulty-conditioned query shows no gain over a fixed query on CONDA, where difficulty is known exactly, the core claim about separating difficulty from memorization is falsified, and the method reduces to learned aggregation.
References
- Detecting Data Contamination in LLMs via In-Context Learning
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Confounders in Instance Variation for the Analysis of Data Contamination
- A Narrowing Geometry in Contaminated Reasoning
- Improving Distantly Supervised Relation Extraction with Neural Noise Converter and Conditional Optimal Selector
- Incorporating Relation Paths in Neural Relation Extraction
- Multi-level Attention-Based Neural Networks for Distant Supervised Relation Extraction
- Neural Knowledge Acquisition via Mutual Attention Between Knowledge Graph and Text
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation