When a language model has memorized a benchmark question, almost any change to that question breaks the memorized shortcut. Once the shortcut is broken, further changes cost little more, because the model is already working from scratch. When a model actually reasons through a problem, each change instead adds its own small difficulty, and these difficulties stack up. This proposal turns that asymmetry into a contamination detector. For each test item we apply every combination of a few meaning-preserving edits, such as renaming quantities, changing an irrelevant detail, paraphrasing, or reordering sentences. We then measure the interaction between edits: whether applying two edits together hurts less than the sum of their separate effects. Memorization should produce a strongly sub-additive, "saturating" pattern, where the first edit destroys the advantage and later edits add nothing. Genuine skill should look roughly additive.
This matters because current detectors are failing exactly where they are needed. Detectors that check how likely the model finds the test text work well after supervised fine-tuning. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows they collapse after one epoch of clean reinforcement learning (RL): LiRA starts at 89.1% AUROC and drops by about 14 points, and reference-free detectors fall to about 50%. The cause is the clipping and importance-sampling steps in GRPO, a common RL algorithm for reasoning models, which squeeze the likelihood gap between seen and unseen items. Our signal is behavioral. It reads the shape of the accuracy surface over a small factorial design (an experiment that runs every combination of factor levels), not a single scalar confidence. We therefore expect it to survive the step that erases the likelihood signal.
Core Claim
Let $p_i(S)$ be the probability that the model answers item $i$ correctly when the subset $S$ of $k$ edits is applied, estimated from repeated samples. For a pair of edits $a, b$, the interaction is
$$I_i(a,b) = p_i(\{a,b\}) - p_i(\{a\}) - p_i(\{b\}) + p_i(\emptyset).$$
In the idealized memorized case, $p_i(\emptyset)$ is high and every edited cell sits near a common "from scratch" level $q_i$. Then $I_i(a,b) \approx p_i(\emptyset) - q_i > 0$ for every pair. Under additive degradation, $I$ is near zero, or negative if difficulties compound. We define a saturation score for each item: the average pairwise interaction, plus the higher-order terms obtained by decomposing the variance of $p_i(S)$ across all $2^k$ cells. We normalize it by the same quantity computed on a clean reference model of similar ability, so that items which are simply fragile to paraphrase are not flagged.
The claim is that this score separates contaminated from clean items better than single-axis perturbation checks at matched query budget, and that it keeps this advantage after clean RL post-training.
Relation to Prior Work
Two detectors already use perturbations or sampling, and they supply our ingredients. CAP perturbs items along factual, logical, and semantic axes and compares a Performance Consistency Ratio between training and test sets. It treats each axis as an independent check and works at the dataset level, so it cannot see whether edits interact. We reuse its edit types but cross them. Generalization or Memorization introduces CDD, which flags memorization when repeated samples for a prompt collapse into near-identical outputs. That is a single-prompt property, and its authors do not study RL-trained models. We use their output-diversity measure as a second response variable inside each design cell.
The motivation for crossing factors comes from the sensitivity-analysis literature. Efficient Error Prediction for Differentially Private Algorithms used a $2^k$ factorial design to find that a three-way interaction, invisible to one-factor-at-a-time testing, drove the error. Global Sensitivity Analysis for MAP Inference in Graphical Models and What Can We Learn from Global Sensitivity Analysis of Biochemical Systems? show that simultaneous small perturbations reveal behavior that single sweeps miss. We borrow the design logic, not their exact algorithms. Those algorithms are restricted to small discrete models, whereas we only need to estimate success probabilities over 16 cells.
Three papers constrain the design. The Impact of Post-training on Data Contamination shows contamination can go dormant during continued pre-training and resurface after clean post-training, so we must test at several pipeline stages. The fragility paper's Stage II result is the most serious: contaminating a reasoning model with chain-of-thought solutions inflated pass@1 by 11.7%, yet all 10 detectors stayed at 50–58% AUROC, because the model generalized to similar unseen problems. If that generalization also carries over to perturbed versions of the leaked items, the saturation pattern could vanish. We treat this as the main way the method could fail, not something to argue around. Finally, Sensitivity of Small Language Models to Fine-tuning Data Contamination found threshold-like, non-linear responses to contamination, but only through single-factor sweeps. This supports the expectation that interaction terms carry signal.
Research Program
We build controlled contamination with known ground truth, so that every item has a true label of seen or unseen.
The first test bed uses DyVal, which generates reasoning problems from directed acyclic graphs. Because the generator controls the problem structure, we can produce perturbed versions that are provably equivalent, and we can draw a fresh, never-seen split. Half of a generated pool is injected during supervised fine-tuning, and the other half stays clean.
The second test bed follows the fragility paper's Stage I pipeline. We contaminate Qwen2.5-7B-Instruct by fine-tuning on part of MATH-500, AIME/AMC, and OlympiadBench, then apply one or more epochs of clean GRPO. The third follows its Stage II pipeline: chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B. A fourth arm reuses the pre-training injection recipe from the post-training paper on GSM8K and MBPP, so the method is tested against the dormant-then-resurfacing regime, with GSMPlus and HumanEval as clean controls.
The factorial design has $k=4$ binary edits, giving 16 cells per item:
- number and entity renaming, with answers recomputed;
- a change to an irrelevant detail, following CAP's logical edit;
- paraphrase;
- reordering of independent premises, which DyVal shows can matter by itself.
The ordering effect is exactly why a clean reference model is needed. In DyVal's experiments, reversing the description order cost GPT-4 13.6% on reachability tasks without any contamination involved.
For each cell we draw $n$ samples, recording both correctness and CDD-style edit-distance diversity. We fit a logistic model per item with all interaction terms and summarize it as the reference-normalized saturation score.
A secondary analysis uses the fitted interaction structure to localize what was memorized. If saturation is driven mainly by the renaming factor, the model stored the numbers. If it is driven by paraphrase, the model stored the surface wording. The single-axis studies cannot provide this kind of diagnosis.
Baselines
The naive reference point is chance (50% AUROC), together with raw accuracy on the original item used as a contamination score. Any useful detector must beat both.
The strongest likelihood-based competitor is LiRA, run as in On The Fragility of Benchmark Contamination Detection in Reasoning Models, along with Loss and Min-K%. It is the strongest because it reaches the best reported figure on this exact question: 89.1% AUROC after supervised contamination, before RL. We run it on the same checkpoints and items, using the same clean reference model that our normalization uses. Beating it means matching it within noise before RL and exceeding it by a clear margin after GRPO, where the reported figure falls by about 14 points. In the Stage II regime the reported detector range is 50–58% AUROC, and any reliable result above that range would be new.
The strongest perturbation-based competitors are a CAP-style consistency score and CDD. We compute both from the same generated edits, at the same total number of model queries per item and the same sampling temperature. For CDD, the reported gain is a 21.8–30.2% relative improvement in accuracy and AUC over n-gram and LLM baselines on DETCON. As a sanity check we also report on DETCON's explicit and implicit leak splits. For CAP, no item-level AUROC is reported on Lacuna; the file only reports that CAP correctly flagged models with a 21-question FinEval overlap.
We also include the partial-prompt completion audit from Reasoning or Memorization?. That paper reports Qwen2.5-Math-7B reaching nearly 40% on MATH-500 given only the first 40% of each problem.
Two ablations isolate the key ingredient. The first fits a main-effects-only model to the same 16 cells. The second runs a one-factor-at-a-time design (5 cells) with the query budget redistributed so each cell gets more samples. If either ablation matches the full score, the interaction claim is wrong.
Why not just…?
Why not just use n-gram overlap against the training data? For most models the training corpora are not disclosed, as the survey notes. Overlap also misses rephrased leaks, which is the gap CDD's reported DETCON gains over n-gram baselines rest on. We include n-gram matching only in the controlled arms, where we know the corpus, to confirm it misses the paraphrased injections.
Why not just run single-axis perturbations and average them, as CAP does? This is the real threat. Under pure memorization, each single edit already breaks the shortcut, so one-factor-at-a-time testing can detect it too. The argument for interactions concerns false positives. A clean item that happens to be fragile to one edit looks identical to a memorized item under single-axis checks. The two diverge only when edits are combined: genuine difficulties compound, while memorized advantages saturate. The equal-budget ablation settles this directly. If it matches the full design, we will report that the simpler check suffices and the contribution reduces to the reference-normalization step.
Why not just keep improving likelihood detectors, for example by removing the effect of clipping? The fragility paper shows that removing clipping restores the likelihood gap. But auditors cannot change how a released model was trained, so the signal has to survive whatever pipeline was actually used. The GRPO-epoch sweep tests whether our behavioral signal decays more slowly than LiRA's.
Why not just evaluate on fresh procedurally generated benchmarks like DyVal or LiveBench? Dynamic generation covers mainly formal tasks, and the field still reports MATH, GSM8K, and HumanEval. Auditing those existing scores remains necessary. We use DyVal as the ground-truth test bed, not as a replacement for the audited benchmarks.
Decisive Checks
The method is supported if four things hold.
First, on DyVal and the Stage I pipeline, the saturation score's AUROC beats the main-effects and single-axis ablations at equal budget.
Second, after one and three epochs of clean GRPO, the saturation score's AUROC stays well above LiRA's on the same checkpoints.
Third, in the clean reference model the interaction terms stay near zero, confirming that the normalization removes ordinary item fragility.
Fourth, the most informative result either way is in the Stage II chain-of-thought arm. If generalization lifts the edited cells along with the original, saturation disappears. That would show this leak type converts into transferable skill rather than brittle recall, which itself bears on whether the 11.7% inflation should be called contamination at all.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- CAP: Data Contamination Detection via Consistency Amplification
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- Efficient Error Prediction for Differentially Private Algorithms
- Global Sensitivity Analysis for MAP Inference in Graphical Models
- What Can We Learn from Global Sensitivity Analysis of Biochemical Systems?
- The Impact of Post-training on Data Contamination
- Sensitivity of Small Language Models to Fine-tuning Data Contamination
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
- Benchmark Data Contamination of Large Language Models: A Survey