When a language model has seen a benchmark's test questions during training, its score on that benchmark goes up for the wrong reason. The usual way to catch this checks whether the model is unusually confident on test items, using the probability it assigns to the text. That signal is fragile. A short round of reinforcement-learning post-training erases it: on On The Fragility of Benchmark Contamination Detection in Reasoning Models, likelihood detectors fall from 89.1% AUROC to near chance, and to 50–58% AUROC when chain-of-thought solutions are leaked directly. We propose a different witness. For each test question, we ask how well the model does on other questions that are verifiably clean and that sit close to it in an embedding space. A kernel-weighted average of those neighbors' outcomes predicts how the model should perform on the test question if it is reasoning rather than recalling. Contamination then shows up as a residual: the model succeeds where its clean neighborhood says it should fail.
The tool for this prediction is a Nadaraya-Watson head. This is a classifier with no output weights: its prediction is a similarity-weighted average of the labels of stored examples. Here the "labels" are the target model's own success rates on clean support questions. The approach needs no log-probabilities, no pool of clean reference models, and no rewriting of the benchmark. It gives a score for every item, together with the specific clean neighbors that justify the score. The central bet is that memorization is local. A leaked item becomes an isolated spike of competence, while genuine ability changes smoothly over nearby problems. That is the picture On the Geometry of Memorization draws for representation curvature on small networks, and a local estimator is the natural way to expose such spikes.
Core Claim
Let $a(x)\in[0,1]$ be the target model's pass rate on test item $x$, estimated from a few samples. Let $\mathcal{S}=\{(s_j,a_j)\}$ be a support set of clean items $s_j$ with the same model's pass rates $a_j$. Let $f$ be an encoder and $K$ a similarity kernel. The neighborhood-predicted pass rate and the contamination residual are
$$\hat a(x)=\frac{\sum_j K(f(x),f(s_j))\,a_j}{\sum_j K(f(x),f(s_j))},\qquad r(x)=a(x)-\hat a(x).$$
Large positive $r(x)$ flags item-level contamination. The mean of $r$ over a benchmark estimates the score inflation. On clean held-out items this mean should be near zero, and that gives the method a built-in calibration check.
The claim is twofold. First, $r(x)$ survives reinforcement-learning "cloaking," because the cloaking result concerns the log-probability gap, not the behavioral advantage itself. In the same Fragility paper, contamination still inflates pass@1 by an average of 8.8% after cleaning by Group Relative Policy Optimization (GRPO). Second, the residual is only trustworthy if the neighborhood ignores where a question came from. A benchmark's house style should not change which clean items count as its neighbors. We get this property by training the encoder with the environment-conditioned support sampling of Learning Invariant Representations with a Nonparametric Nadaraya-Watson Head. That paper draws the support set from a different environment than the query, which forces distances to reflect content rather than source. It used this to reach 80.0% on Camelyon-17 versus 70.3% for standard training. Here the environments are benchmark sources.
Relation to Prior Work
Two papers motivate the gap. The Fragility paper shows that likelihood detectors break after RL. Detecting RLVR Training Data via Structural Convergence of Reasoning responds by sampling 32 completions per prompt and measuring nearest-neighbor edit distance among them. Seen prompts produce near-identical reasoning, and the method reaches an average AUC of 0.70. That method uses nearest neighbors among one prompt's own outputs, and it targets RL-trained math and code. We use nearest neighbors among different, clean prompts. Our method also applies to any task with a checkable answer, including multiple choice.
ConStat is the closest relative in spirit. It flags a model whose benchmark score exceeds what a difficulty curve fitted on about 20 clean reference models predicts. It reports, for example, δ≈8.3% for Mistral-7B on GSM8K and δ≈6.6% for Llama-3-70B on ARC-Challenge. ConStat works at the level of whole benchmarks and needs clean models. We replace clean models with clean items evaluated on the same model. That substitution moves the analysis from benchmark-level to item-level and removes the "whole model family is contaminated" failure that ConStat's authors acknowledge.
CoDeC measures whether adding in-context examples lowers likelihood, which happens for memorized items. It reaches 99.9% AUC at the dataset level. It needs log-probabilities and is weakest at the sample level, which is exactly where we aim.
Two papers constrain the design. The Emperor's New Clothes in Benchmarking? shows that paraphrased variants do not stop contaminated models from scoring well, while aggressive rewrites drop fidelity to 0.686 on MMLU. Two consequences follow for us. Our support items must be distinct problems, not paraphrases of test items. And we never modify the test items at all.
The method borrows from three sources. The estimator and its environment conditioning come from the Nadaraya-Watson papers, including A Simple Nadaraya-Watson Head for Explainable and Calibrated Classification, which also motivates using the retrieved neighbors as explanations. The idea of reading a model's behavior through a similarity datastore rather than its output head comes from $k$NN Prompting. That paper stores label-paired output distributions and gains up to +18.84 points over in-context learning, but it never asks what leaked anchors do. Its datastore idea becomes our support store.
Research Program
We would build three components.
The first is a clean support pool for each domain. Clean items come from post-cutoff sources such as LiveBench, procedurally generated items such as A²RBench and the Cognitive Battery, and, in controlled experiments, held-out sibling benchmarks. For competition math, the target benchmark is AIME or AMC, and the support comes from MATH 500, OlympiadBench, Minerva Math, and LiveBench math.
The second is the encoder. We compare three choices. One is a frozen general-purpose text encoder, which is fully black-box. Another is the target model's own intermediate hidden states, which is gray-box. This choice is interesting in its own right, since the memorization-geometry work predicts that leaked items may move to unusual positions in representation space. The third is a small encoder trained with an implicit Nadaraya-Watson objective. In this objective, query and support are always drawn from different sources, and the training target is a clean model's pass rate. This teaches the encoder to place problems by difficulty-relevant content rather than by benchmark of origin.
The third is the audit procedure. We estimate $a(x)$ with a few samples per item, compute $r(x)$, and then compute a benchmark-level inflation estimate with bootstrap intervals. We set the threshold using the requirement that $r$ averages to zero on items known to be clean.
One limitation applies to every behavioral detector. If a model would answer an item correctly anyway, leaking that item changes nothing and cannot be detected. We therefore report item-level detection both overall and restricted to items the uncontaminated base model fails. We treat inflation estimation as the primary target.
Baselines
The naive reference point is raw benchmark accuracy treated as uncontaminated, together with the plain Loss and Min-K% likelihood scores. These show the size of the problem. The Fragility paper reports that Loss detection drops by about 16% after one GRPO epoch.
The strongest likelihood competitor is LiRA. It reports 89.1% AUROC on supervised-fine-tuning contamination before RL, and that is the number we must approach after RL, where LiRA itself degrades. We run it on the same model checkpoints.
The strongest RL-specific competitor is Min-$k$NN Distance from the structural-convergence paper, with a reported average AUC of 0.70. It is run with its own $N=32$ completions. For a fair comparison, we also give our method a matched total sampling budget, counting support-pool evaluation, and report results at matched budgets.
CoDeC is the strongest dataset-level detector, at 99.9% AUC dataset-level. We run it wherever log-probabilities are available. We did not find a reported sample-level number for CoDeC on Lacuna, so we measure it ourselves on our splits.
ConStat is the strongest black-box, performance-based competitor. We run it with the authors' protocol, compare its benchmark-level inflation estimates to ours, and in the controlled setting compare both to the true injected inflation.
Two ablations isolate the key ingredient. In the first, we replace the Nadaraya-Watson estimator with a parametric difficulty regressor (ridge and a small multilayer perceptron) on the same embeddings and the same support. In the second, we keep the estimator but drop environment conditioning, sampling support from the same source as the query. A further control replaces the support with randomly chosen clean items, which tests whether neighborhood structure matters at all.
Beating the field means three things. Item-level AUROC must clearly exceed 0.70 and stay well above the 50–58% likelihood range in the chain-of-thought contamination setting. Inflation estimates must land within the bootstrap interval of the true injected inflation. And the ablations must show measurable losses.
Why not just…?
Why not just use ConStat? ConStat answers a different question: whether a model's total score is suspicious relative to other models. It cannot say which items are leaked. It also needs clean reference models, which its authors concede may not exist if a whole generation is contaminated. That concern is sharpened by its own finding that the top-3 Open LLM Leaderboard models each showed more than 10% inflation. Still, at the benchmark level ConStat is a real threat. If its inflation estimates match ours in the controlled setting, our contribution narrows to item-level resolution and independence from reference models, and we will report it that way.
Why not just compare scores on the benchmark and a paraphrased copy? The Emperor's New Clothes paper found that surface rewrites keep fidelity above 0.9 but give near-zero resistance, because contaminated models map paraphrases back to memorized answers. A paraphrase gap therefore underestimates contamination. Our support items are different problems, not rewrites, so the memorized answer does not transfer. We include the paraphrase gap as a baseline to show this directly.
Why not just train a parametric difficulty predictor instead of a Nadaraya-Watson head? This is the most serious alternative, and the ridge and multilayer-perceptron ablation exists to settle it. Our reasons to expect a gap are as follows. A global regressor smooths over exactly the local spikes we are looking for. It cannot point to which neighbors justify a flag. And it inherits shifts between benchmark styles unless something like environment-conditioned support forces invariance. If the regressor matches our estimator on the same encoder, we will conclude that the clean-support residual, not non-parametric inference, is what matters, and reframe the work accordingly.
Why not just use CoDeC, given its 99.9% AUC? That figure is at the dataset level and assumes access to log-probabilities. Our method targets black-box APIs and item-level decisions, where CoDeC's own authors note weakness at ambiguous boundaries. Where both methods can run, we report both and also test whether combining them helps.
Proposed Experiments
Controlled contamination following the Fragility protocol. In Stage I, we fine-tune on half of AIME, AMC, OlympiadBench, and Minerva Math, then clean with GRPO. In Stage II, we inject chain-of-thought contamination into DeepSeek-R1-Distill-Llama-8B, the setting where the Fragility paper reports an 11.7% boost and 50–58% detector AUROC. The other half of each benchmark serves as clean negatives. We compute residuals for the base checkpoint and the contaminated checkpoint, and use the change in residual to cancel any contamination already present in pretraining. The decisive check is whether item-level AUROC on the RL-cleaned checkpoints exceeds both likelihood detectors and Min-$k$NN Distance. A second check is whether the mean residual recovers the 8.8% average inflation that the Fragility paper reports for Stage I.
Multiple-choice transfer. We contaminate a 7B model on half of MMLU and ARC-Challenge, with support drawn from other multiple-choice sources and RepliQA-style fictional-entity items. This tests whether the method works beyond RL-trained reasoning, where Min-$k$NN does not apply.
Natural audit. We apply the frozen-encoder variant to open models on GSM8K, ARC-Challenge, and Hellaswag. We check whether its inflation estimates agree in direction with ConStat's reported gaps for Mistral-7B and Llama-3-70B. Agreement is a sanity check, not ground truth. We then inspect the top-flagged items and their retrieved neighbors.
Falsification test. If, after RL, the residual distribution on leaked items does not separate from that on clean items, then memorization is not local in any embedding we can build. That would mean RL-era contamination needs a non-behavioral signal, and the program would stop there.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- On the Geometry of Memorization: Interpolation and Second-Order Representation Irregularity
- Learning Invariant Representations with a Nonparametric Nadaraya-Watson Head
- Detecting RLVR Training Data via Structural Convergence of Reasoning
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Detecting Data Contamination in LLMs via In-Context Learning
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
- A Simple Nadaraya-Watson Head for Explainable and Calibrated Classification
- $k$NN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- A Cognitive Battery for Foundation Models: Theory-Grounded Benchmarks for Attention, Learning, Metacognition, Executive Function, and Social Cognition