Language models score highly on public benchmarks, and two separate literatures warn us not to trust those scores. One shows that much benchmark performance may come from having seen the test items during training, which is called contamination. The other shows that inserting a single well-chosen distracting sentence can cut accuracy by half. These literatures have never been tested against each other. Nobody knows whether a memorized test item is harder or easier to knock over with irrelevant context than an item the model actually reasons through. This proposal treats that unanswered question as a measurement opportunity. We add irrelevant context to each item in small, controlled amounts that leave the correct answer unchanged, and we record how accuracy falls as the amount grows. Our bet is that the shape of this "dose-response" curve differs between memorized and genuinely solved items. If it does, the curve becomes a per-item contamination test that needs only the model's answers, not its internal probabilities.
The test matters most where current tools reach least. The strongest existing detector works on whole datasets and needs the model's token probabilities. It has never been run on agent benchmarks, where a model acts over many turns inside a software environment. Agent benchmarks built from public data, such as Kaggle notebooks, are exactly the ones future training runs will absorb. Our approach uses that agent setting as a second test: we put look-alike decoy files into a data-science sandbox and check whether the same fingerprint appears.
Core Claim
The core claim is that two kinds of distracting context should affect memorized and genuinely solved items in opposite directions. We call these "surface-preserving" noise, which leaves the original question text intact and adds irrelevant material around it, and "surface-breaking" noise, which disrupts the familiar token sequence while keeping its meaning. Contamination should show up as a sign flip between the two dose-response slopes, not as a lower or higher accuracy level.
The motivation comes from a direct tension in the file. Token-level surprise studies find that classic puzzles are predicted with near-certainty (Yax et al.). That suggests memorized recall is keyed to the familiar string and should survive surrounding noise.
CoDeC points the other way. It shows that prepending same-dataset examples lowers the model's confidence on memorized data but raises it on unseen data. That suggests memorized items sit in a narrow basin that extra context knocks them out of.
These two intuitions make different predictions for different noise types, so they can be told apart experimentally. The file states plainly that this has "never been tested side-by-side."
What is new relative to CoDeC is threefold:
- Many noise types at graded strength. CoDeC uses one kind of context (random in-context examples) at one dose. We use several noise types at graded strength, with the answer held fixed.
- Accuracy only. CoDeC reads log-likelihoods. We read only whether the answer is correct, so the probe works on closed models and on agents.
- Per-item scores. CoDeC is reliable as a dataset-level aggregate. Our target is item-level scoring.
What is new relative to Adaptive Distraction and Breaking Focus is the goal. Those papers search for the single most damaging appended sentence in order to attack the model. We want calibrated, graded noise that measures something about the item.
Research Program
The program has three pieces: a noise-and-twin generator, a ground-truth contamination testbed, and the probe itself.
The generator. This is where multi-agent task synthesis earns its place. Hand-written distractors are unusable here because the probe is worthless if a distractor silently changes the correct answer. AUTOPLAY reports that single automated verifiers agree with humans only about 60% of the time, with an optimistic bias. We therefore borrow the assembly-line structure of ORACLE, whose environment builders, simulated testers, and debugging agents check one another, and the generator-versus-discriminator-ensemble design of CoDA-Bench. The pipeline has three agents:
- A noise writer produces distractors at five strength levels for each noise type.
- An invariance checker re-solves the noised item with a strong model and, where possible, a programmatic solver, and rejects any distractor that changes the gold answer.
- A twin builder produces a meaning-preserving rewrite of each item. It follows BeyondBench for algorithmic items, using solver-backed isomorphic transforms, and follows the solution-anchored back-construction of CoDA-Bench for data tasks.
The noise types. Surface-preserving noise includes appended irrelevant sentences in the style of error-guided perturbation, extra unrelated passages, and, in the agent setting, look-alike distractor files such as sales_2022.csv beside sales_final.csv. Surface-breaking noise includes interleaved unrelated in-context demonstrations, which is CoDeC's own perturbation generalized to graded doses, plus reordering of option or clause structure and the twin rewrite itself.
The ground-truth testbed. We need items whose contamination status we know by construction, not by inference. We use open models trained on known corpora: the Pythia family (trained on The Pile) and a Dolma-trained model. CoDeC already shows these corpora support clean seen-versus-unseen labels. On top of this we run controlled continued pretraining in which a random half of a benchmark split is injected at chosen repetition counts. This gives item-level labels and a dial for how strongly an item is memorized. For clean references we use TEMPORA-0325, documents published after March 2025 and turned into questions via YourBench, and freshly sampled BeyondBench instances.
The probe. For each item we fit two slopes, the change in accuracy per noise level under surface-preserving and under surface-breaking noise, estimated from repeated sampling at each level. A small classifier maps the slope pair to a contamination score. We calibrate it once on the injection testbed and then apply it frozen to new models and benchmarks.
A secondary mechanistic analysis. On the open models we extract task vectors at the intermediate layer that In-Context Learning Creates Task Vectors identifies, which is where demonstrations are compressed into a single steering state. We ask whether surface-breaking noise shrinks or rotates that vector more for memorized items. The task-vector paper verified the mechanism only on simple single-token tasks, so this analysis is confined to the multiple-choice settings where that holds. We treat it as explanation, not as a required part of the probe.
Baselines
Clean accuracy (the naive reference). This is accuracy on the unperturbed item, used as a contamination score: suspiciously high accuracy is taken to mean memorized. It is the "do nothing" point. It should fail, because clean accuracy confounds memorization with ease. The probe must beat it by a wide margin in item-level area under the ROC curve (AUC) on the injection testbed.
CoDeC. This is the strongest existing method and the most direct competitor: it computes the share of samples whose log-likelihood drops when same-dataset examples are prepended. It reports a dataset-level AUC of 99.9% separating seen corpora from unseen benchmarks (CoDeC). We found no reported sample-level result on Lacuna for CoDeC.
We run CoDeC on the same open models, the same items, and the same number of forward passes per item as our probe. Fairness requires this matched budget because our probe samples each item many times.
We do not expect to beat 99.9% at the dataset level. Beating CoDeC means two things:
- higher item-level AUC on the injection testbed at matched compute, and
- usable detection where CoDeC cannot run at all, namely closed models answering through an interface and multi-turn agents.
If CoDeC's per-item confidence drop already matches our item-level AUC on open models, then our contribution narrows to the black-box and agent settings. We will report it that way.
Likelihood-based detectors. Min-K% and Zlib-ratio detectors are included because reviewers expect them. CoDeC reports outperforming them, but we found no reported numerical result on Lacuna for Min-K% or Zlib on these benchmarks, so we run them ourselves on the same splits.
Twin gap. This is accuracy on the original item minus accuracy on its meaning-preserving rewrite, which is the logic behind BeyondBench's isomorphic transforms. It is the most serious simple competitor. It gets identical twins and sampling budget.
Single-slope ablations. These isolate the key ingredient. One probe uses only the surface-preserving slope and another uses only the surface-breaking slope. The core claim predicts that the slope pair outperforms either slope alone. If one slope alone matches the pair, the "opposite directions" story is wrong even if detection works.
Why not just…?
Why not just use the twin gap, since rewriting an item and checking for an accuracy drop already detects memorization? This is a real threat, and we test it head-on rather than argue around it. The twin gap is one point on our surface-breaking curve. Our claim is that the surface-preserving slope adds independent information, because the twin gap confounds memorization with any difficulty shift the rewrite introduces. BeyondBench's observation that models above 90% on GSM8K "often see their accuracy cut in half (or worse) on uncontaminated tasks" cannot by itself separate those two causes. The single-slope ablation against the full probe on the injection testbed settles the question. If the twin gap matches the full probe, we drop the detector claim and report the Gap 4 finding as the result, together with the twin generator.
Why not just run CoDeC everywhere? CoDeC needs token log-likelihoods and is reliable at the dataset level. Neither condition holds for closed agent systems such as the Mini-SWE-agent setup evaluated on CoDA-Bench, where the unit of interest is a single multi-turn trajectory. The research file lists extending CoDeC to dynamic agent traces as entirely untested. Our agent experiment is the direct check of whether an answer-only probe can fill that gap.
Why not just tell the model to ignore irrelevant context, and treat whatever drop remains as pure contamination signal? Adaptive Distraction reports that such instructions "provided almost no defensive benefit." Prompting therefore neither removes the noise effect nor isolates it. We include the instruction as a fixed condition in every run so it cannot explain differences between memorized and clean items.
Why not just ask one LLM to write the distractors? If a distractor changes the gold answer, an accuracy drop no longer means sensitivity to noise. AUTOPLAY's roughly 60% verifier agreement shows that unchecked single-model judgment fails often enough to swamp the effect we want to measure. We report the rejection rate of the invariance checker and the human-audited validity of a sample of accepted distractors. If the checker rejects almost nothing, the single-agent alternative was sufficient and we say so.
Proposed Experiments
Experiment 1: the core sign-flip test. This runs on GSM8K, MMLU, CommonsenseQA, and MATH-500, using the injection testbed and the known-corpus models. For each injection repetition count we plot both slopes for injected and held-out halves. The decisive check is whether the sign of the slope difference separates the halves and grows with repetition count. Item-level AUC is the metric. A null result here, with identical curves for injected and clean items, is itself publishable: it resolves Gap 4 in favor of "contamination does not change how models break."
Experiment 2: transfer to new models and benchmarks. We freeze the calibrated probe and apply it to API models on the same benchmarks, on TEMPORA-0325 questions as a known-clean set, and on BeyondBench instances as a second clean set.
- Success criterion: a low contamination score on both clean sets and a high score on classic benchmarks.
- Anchor: the reported CommonsenseQA collapse for GPT-4o-mini under adaptive distraction, from 85.7% to 22.0% (Breaking Focus).
- Check: we test whether our graded surface-preserving noise reproduces a drop of this order at its strongest level. This tests whether calibrated noise is strong enough to matter.
Experiment 3: the agent setting on CoDA-Bench. CoDA-Bench reports 61.1% execution accuracy for Mini-SWE-agent with GPT-5.5 on the full set and 49.6% on CoDA-HARD. It also reports gains of nearly 25–28% when the agent is told the exact target files. That oracle-versus-full gap is a ready-made two-point dose curve.
We densify it by varying the number of verified look-alike decoy files from zero up to the benchmark's natural level of about 980 files per environment. As surface-breaking noise we use our twin builder's re-anchored versions of the task. The hypothesis is that tasks whose Kaggle source predates the model's training cutoff show flat curves under decoys: the answer is recalled regardless of which file is opened. Tasks built from post-cutoff data should show steep curves.
The decisive check has two parts. First, these curves should agree with the twin gap on the same tasks. Second, we inspect trajectories for answers produced without ever opening the correct file, which is direct behavioral evidence of recall.
Experiment 4: mechanism. On the open models we measure the norm and direction of the task vector as surface-breaking dose increases, separately for injected and held-out items. Like Experiment 1, this is decisive only if an effect appears. If task vectors behave the same for both groups, we report that the behavioral fingerprint has a different mechanism than demonstration compression.
References
- Studying and improving reasoning in humans and machines
- Detecting Data Contamination in LLMs via In-Context Learning
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Scaling Synthetic Task Generation for Agents via Exploration
- Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interaction
- CoDA-Bench: Can Code Agents Handle Data-Intensive Tasks?
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- Yourbench: Dynamic Evaluation Set Generation with LLMs
- In-Context Learning Creates Task Vectors