Contamination Along the Abstraction Ladder

When a benchmark question leaks into a model's training data, today's audits give a yes-or-no answer. They ask whether the model has seen the item, usually by checking text overlap or whether the model assigns suspiciously high probability to it. That question has become both hard to answer and the wrong one to ask. Reinforcement learning after contamination erases the probability signal: likelihood-based detectors fall to about 50% AUROC even though accuracy stays inflated (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Memorizing an item also does not by itself mean the model can exploit it (Data Contamination: From Memorization to Exploitation). What an evaluator actually needs to know is how far the leaked advantage reaches. Does it help only on the exact item, on the same problem with new numbers, or on the whole family of problems that share its logic?

We propose to measure contamination as a curve rather than a bit. We borrow the machinery of specific-to-general rule induction, the older tradition of learning rules bottom-up by treating each training example as a maximally specific rule and then loosening it step by step (Rule Induction and Instance-Based Learning: A Unified Approach). We run that machinery in reverse as a probe generator. Each suspect benchmark item becomes the bottom rung of a ladder. Each higher rung is produced by one bottom-up operation: stripping context, relaxing constants, or taking the minimal generalization shared with a sibling item. Every rung has an answer computed by executing a program. The model's accuracy advantage on seen items, traced up the ladder, gives what we call an exploitation profile. The highest rung where the advantage survives is the leak's generalization radius. Because the probe is purely behavioral, it needs no weights or likelihoods, so it should survive the reinforcement-learning "cloaking" that defeats current detectors.

Core Claim

The claim has two parts. First, contamination leaves a characteristic shape of accuracy advantage across graded, rule-structured variants of a leaked item. This shape persists after clean reinforcement learning has flattened the log-probability gap, because the fragility study reports that pass@1 inflation (8.8% on average in its Stage I) survives that training. Second, the shape tells different kinds of contamination apart, and a single perturbation cannot do this. Verbatim question-answer leakage should produce an advantage that collapses at the first surface or constant relaxation. Leakage of chain-of-thought solutions should carry the advantage up through constant relaxation and fade at the schema level. Genuine skill should produce a flat profile that matches never-published control items. If these profiles are separable, auditors gain two things: a detector that works in the regime where current detectors fail, and a measurement of whether a leak was actually exploitable beyond the item.

Relation to Prior Work

Three lines of work motivate the proposal. The fragility study documents detector collapse under reinforcement learning and chain-of-thought contamination. Test of Time and Beyond Memorization show that rephrasing problems into multi-step reasoning removes post-cutoff decay and drops influence-function hit rates from 77.5% to 17.5%. That result cuts two ways. Transformation can hide leaks, and it can also serve as a measuring instrument. We use it as an instrument, applied in controlled, graded steps rather than as one rewrite. Data Contamination: From Memorization to Exploitation separated memorization from exploitation for encoder models on classification tasks. We extend that distinction from a single seen/unseen contrast to a continuum of abstraction, and to generative reasoning models.

Three dynamic-benchmark efforts are the closest existing work, and each differs from ours in a specific way. DyVal generates fresh reasoning tasks from random graphs, but its items bear no relation to any particular leaked item. It can tell you a model is weaker on new material, but it cannot trace an advantage back to a specific leak. A²RBench has an author model invent transformation rules top-down, and adds a single symbolic-remapping test, on which models drop 17.7%. That remapping is one rung of our ladder, not a ladder, and the author model's difficulty ceiling does not apply to us because our variants are derived from existing seed items. CellARC supplies something we reuse: an oracle that labels each query cell of a cellular-automaton task as recoverable by lookup, recoverable only via the rule, or ambiguous. This gives an exact ladder in a synthetic domain.

The rung-building operators come from the rule-induction papers. Context stripping follows Identifying Correction Rules for Auto Editing. Pairwise minimal generalization, which finds the least general pattern that covers two instances performing the same change, follows Were We There Already?. Two prior results constrain the design. That same paper's rule counts ("thousands to tens of thousands" per language) and the slowdown of interval-based rule extraction (Extracting Rules from Artificial Neural Networks with Distributed Representations) warn that pairwise generalization explodes combinatorially. We therefore only generalize within a suspect item's small neighborhood. When Is Benchmark Contamination Detectable? shows that audit power scales with contamination fraction times separability times the square root of sample size, and that naive sample-size formulas fail below 50 items. We use its simulation planner to set sample sizes rather than guessing.

Research Program

The first component is the ladder builder. For math items, each seed problem is converted into an executable solution program, which is accepted only if executing it reproduces the gold answer. Rungs are then produced by one operation each. Rung 0 is the verbatim item. Rung 1 relaxes context: entities are renamed and the problem is reworded while the program stays fixed. Rung 2 relaxes constants: the same program runs on new numbers. Rung 3 is minimal generalization. The seed is paired with another item that uses the same sequence of operations, the constraints the two do not share are dropped, and a new item is instantiated from the resulting schema. Rung 4 relaxes structure: the schema is merged with a sibling that differs by one step, and an item is drawn from the merged schema. Answers at every rung come from execution, so no model ever needs to solve a problem in order to author it.

In the cellular-automaton domain, the rungs are exact:

  • the same support set and query;
  • new supports from the same rule, with queries restricted to lookup-identifiable cells;
  • the same rule with queries restricted to rule-identifiable cells;
  • a neighboring rule in the same family.

The second component is the statistic. In controlled experiments, where we know which items leaked, the exploitation profile at rung $k$ is $E_k = a_k^{\text{seen}} - a_k^{\text{unseen}}$. Here $a_k$ is accuracy on rung-$k$ variants of seen or unseen items for the same model. In black-box audits of models whose training data is unknown, we cannot mark any items as unseen. Instead we compare how steeply accuracy falls from rung 0 to rung $k$ on suspect items against the same drop on freshly authored, never-published control items built from the same schemas. This is a difference-in-differences. It cancels the ordinary difficulty change between rungs because controls climb the same ladder. The generalization radius is the largest $k$ at which $E_k$, or the black-box difference, is significantly above zero.

Baselines

The naive reference point is to report raw benchmark accuracy with no audit. This is what leaderboards do, and it anchors how much inflation we are trying to explain.

The strongest existing detector in our setting is LiRA, a likelihood-ratio membership test. We rerun it in the Stage I and Stage II protocols of the fragility paper. That paper reports 89.1% AUROC for LiRA before clean GRPO training (a reinforcement-learning method), a drop of about 14 points after one epoch of it, and 50–58% AUROC for all ten detectors after chain-of-thought contamination. We also run Min-K% and plain loss thresholds. Beating LiRA means clearly exceeding its post-reinforcement-learning AUROC on the same contaminated checkpoints and item splits, while staying competitive with its 89.1% before reinforcement learning. Our method sees only sampled answers, never log-probabilities.

CDD, the output-dispersion detector from Generalization or Memorization, is the strongest black-box competitor. That paper reports relative gains of 21.8–30.2% over earlier baselines, but no absolute AUROC on our reasoning-model setting appears on Lacuna. We therefore run CDD ourselves, giving it the same generation budget as the ladder: the same total number of sampled tokens per item.

The key ablation is a single-rung audit. It keeps only rung 1 or only rung 2, mirroring HumanEval_R-style rephrasing, variable renaming, or A²RBench remapping. It gets the same total query budget by using more samples per rung.

A second ablation replaces the ordered ladder with the same number of random perturbations matched in edit distance. This isolates whether the bottom-up ordering by abstraction matters, or merely having many perturbations.

All methods get identical items, models, sample budgets, and a fixed tuning set for thresholds. Sample sizes come from the power-calibrated planner.

Why not just…?

Why not just run n-gram overlap filtering? It cannot see rephrased or transformed leaks. The survey and the CDD paper both report that it misses implicit contamination. It also answers "was the text present," not "was the advantage exploitable." We include it only as a sanity floor.

Why not just use one paraphrase, as HumanEval_R did? This is the real threat. The 41.5% leak ratio reported for ChatGPT on HumanEval shows a single rephrasing already exposes a lot, and for pure detection AUROC one well-chosen rung may match the full ladder. If the single-rung ablation matches the ladder on detection, we will say so. The contribution then narrows to what one rung cannot provide: telling verbatim leaks from chain-of-thought leaks, and estimating the radius. The decisive test is whether the full profile predicts held-out transfer, meaning the advantage on sibling items never shown during probing, better than the single-rung gap.

Why not just evaluate on a fresh dynamic benchmark like DyVal? Fresh benchmarks estimate uncontaminated skill but say nothing about which published items leaked or how far a leak's advantage extends. DyVal's own finding that models strong on static benchmarks score near zero on it shows the size of the gap, not its cause. Our control items play exactly this role inside the ladder.

Why not just inspect internal geometry, as in the narrowing-geometry work? A Narrowing Geometry in Contaminated Reasoning needs weights and was shown only on arithmetic. For open models we will report its eigenspectrum index alongside our profile as a cross-check, but it cannot audit closed models.

Proposed Experiments

The first experiment is a controlled cellular-automaton study. On CellARC-100k we train small transformers from scratch, following CellARC's setup, where a vanilla transformer reaches 51% on the interpolation split and 29% on the extrapolation split. We inject a subset of test episodes into training at duplication levels from 1× to 200×; the upper end is chosen because the memorization-versus-exploitation study saw exploitation reach 40% at 200 duplications. Because the oracle labels are exact, we can check whether $E_k$ is concentrated on lookup-identifiable cells, as the radius hypothesis predicts.

The second experiment is a controlled reasoning-model study. Using DeepSeek-R1-Distill-Llama-8B and the fragility paper's two-stage protocol on AIME, AMC, Minerva Math, and GSM8K, we contaminate with question-answer pairs or with chain-of-thought solutions, then apply clean GRPO training. For every checkpoint we compare ladder AUROC against LiRA, Min-K%, loss, and CDD. We also estimate the radius under each contamination type.

The third experiment is a black-box application to HumanEval and GSM8K on API models, using the difference-in-differences statistic against freshly authored controls.

The decisive checks are three. First, the ladder statistic must beat LiRA after reinforcement learning, where LiRA sits near 50%. Second, the full profile must beat both the single-rung and random-perturbation ablations at predicting held-out sibling transfer. Third, question-answer and chain-of-thought contamination must produce distinguishable radii. If clean reinforcement learning flattens the behavioral profile as thoroughly as it flattens likelihoods, the claim fails. That outcome would imply the retained inflation had already generalized beyond the leaked items, which is itself a result worth reporting.