When a language model has trained on a benchmark question, its score on that question no longer measures reasoning. The usual way to catch this is to look at what the model outputs, for example how confident it is in the exact answer text. That signal is now breaking down. Reinforcement learning after pretraining, and training on full worked solutions, both erase the output-level fingerprints of memorization while leaving the inflated score in place. This note proposes looking inside the model instead. The idea is borrowed from 3D graphics. There, a model that represents a face or body tends to let who the subject is leak into what the subject is doing, so that changing an expression also warps the face. We argue that a contaminated language model shows the same flaw. Its internal state for a test question carries an item-specific "identity" component, a trace of having seen this particular question. That trace persists when the question is reworded and even when its logic is changed. A model that is merely good at the task has no such component.
Concretely, for each benchmark item we generate two kinds of controlled variants. Rewordings keep the logic and change the surface. Logic edits keep the surface and change the answer. We then fit a small disentangling model to the deep-layer activations of known-clean calibration items. It separates a "surface" factor from a "logic" factor, using the swap-and-reconstruct training that 3D shape/pose work uses. Whatever a test item's activations share across all of its variants, after surface and logic are explained away, is its identity residual. The contamination score is how consistent that residual is. The method needs one forward pass per variant and no sampling, and it does not assume the model's outputs collapse onto a memorized string.
Core Claim
Contamination leaves an item-specific component in a model's hidden states that is invariant to both rewording and logic edits. This component survives reinforcement-learning post-training and chain-of-thought contamination, the two regimes where output-based detectors fall to near chance. Factoring hidden states into surface and logic subspaces, then scoring the cross-variant consistency of what remains, should detect it at the level of individual items.
Why this gap is real
The motivating evidence comes from On The Fragility of Benchmark Contamination Detection in Reasoning Models. That paper contaminates Qwen2.5-7B-Instruct through supervised fine-tuning, then applies clean GRPO, a group-based policy-gradient method with PPO-style clipping. It shows that the clipping shrinks the gap in log-probability between seen and unseen items. LiRA falls from 89.1% AUROC by about 14 points, and reference-free detectors drift toward 50%. When DeepSeek-R1-Distill-Llama-8B is contaminated with chain-of-thought solutions, its benchmark score inflates by 11.7%, while all ten detectors sit at 50–58% AUROC. The paper explicitly offers no representation-level alternative. The detection problem is therefore open precisely where modern models live.
Two papers suggest the signal survives internally. Think or Remember? finds neurons whose mean activation differs between memorized and computed behavior. These neurons concentrate in deeper layers, which tells us where to look. However, the evidence comes from synthetic bAbI-style and poisoned-arithmetic tasks on GPT-2 and LLaMA 3.2-3B, and the paper never addresses benchmark detection. Generalization or Memorization: Dynamic Decoding for Mode Steering trains a linear probe on intermediate activations to detect reliance on memory. Its labels come from an output-diversity heuristic, and that heuristic is exactly what chain-of-thought contamination defeats, because such models produce diverse rationales. We reuse both papers' localization findings, and we replace diversity-based labels with a variant-consistency signal.
The method itself comes from disentanglement in vision. Unsupervised Shape and Pose Disentanglement for 3D Meshes trains on swapped codes, decoding subject A's shape with subject B's pose, so that neither code can carry the other's information. Mimic adds a gradient-reversal classifier that strips speaker identity out of a content code. Mimic's own diagnostic is that a classifier recovers speaker identity from motion alone with nearly 90% accuracy, and this is the closest analogue to what we want to measure. Both methods construct a clean factorization so that leakage becomes visible as what is left over. We do the same with activations, treating benchmark-item identity as the leaking factor.
On the input side, DyCodeEval and DyVal separate logic from cover story in the prompt text, in order to measure aggregate score inflation. We use the same kind of controlled rewrites as probes rather than as a replacement benchmark, and we ask a different question: which specific items did this model see?
Research Program
The first component is variant construction. For each math item (AIME, AMC, OlympiadBench, Minerva Math, GPQA Diamond, following the Fragility setup), we create about eight rewordings that preserve the answer, using paraphrase and cover-story changes. We also create about eight logic edits that change constants or a single constraint, with answers recomputed by a symbolic solver and checked by execution. For HumanEval, rewordings follow the DyCodeEval scenario-rewrite recipe, and logic edits alter the specification and unit tests. Given the consistency problems DyCodeEval reports for small rewriter models, we verify a sample of variants by hand and drop any item whose variants fail verification.
The second component is the factorization. We take activations at the final prompt token from the deep-layer band that Think or Remember? identifies, chosen per model by activation patching as in DMS. On calibration items known to be clean (problems from competitions held after the training cutoff, in the spirit of MathArena and LiveBench), we train two small encoders, $E_s$ for surface and $E_l$ for logic, and a decoder $D$. The training objective is to reconstruct activations from crossed codes. Writing $h(v)$ for the activation of variant $v$, the decoder must recover $h$ of the variant that shares item $a$'s logic and item $b$'s wording from $D(E_l(a), E_s(b))$. A gradient-reversal classifier stops $E_l$ from encoding which item a variant came from beyond its logic. For any test item $x$ with variants $V(x)$, the residual is $r(v) = h(v) - D(E_l(v), E_s(v))$, and the score is
$$S(x) = \frac{2}{|V(x)|(|V(x)|-1)} \sum_{v \neq v' \in V(x)} \cos\big(r(v), r(v')\big),$$
which is the mean pairwise cosine similarity of residuals across the item's variants. On clean items, residuals should behave like independent noise, so $S$ should stay near the calibration mean. On a memorized item, a shared identity direction should push $S$ up. This should hold even across logic edits, because the item's memorized identity is what the edits fail to change.
The third component is the contamination testbed. We reproduce Stage I and Stage II of the Fragility paper with a known random half of each benchmark contaminated, which gives ground-truth membership labels. We add a rephrased-contamination arm, training on reworded items in the manner of Rethinking Benchmark and Contamination, and a near-neighbor arm, training on problems that are similar to benchmark items but not identical to them.
Baselines
The do-nothing reference is the uncontaminated base model scored with each detector. It fixes the chance level and shows how much score inflation each contamination arm produces. We also report raw model loss as the simplest membership test.
The strongest existing output-level competitors are Min-K% and LiRA, run exactly as in On The Fragility of Benchmark Contamination Detection in Reasoning Models. That paper reports LiRA at 89.1% AUROC after supervised contamination, a drop of about 14 points after one GRPO epoch, and 50–58% AUROC averaged across all ten detectors under chain-of-thought contamination. We rerun these on our own checkpoints rather than copying the figures, so that every detector sees identical models and items. LiRA receives the same reference-model budget the original paper used. We also include CDD from Generalization or Memorization: Data Contamination and Trustworthy Evaluation, which on DETCON reports 21.8–30.2% relative gains in accuracy and AUC over prior baselines. We give CDD a sampling budget at least equal in forward-pass compute to our variant passes.
At the representation level, the competitors are the neuron mean-difference probe from Think or Remember? and the diversity-labeled linear probe from Dynamic Mode Steering, both fitted at the same layers on the same calibration data. We also test a supervised linear probe trained on true membership labels from a different contaminated model and transferred to the target model. This is an upper-bound-style competitor that assumes access to labels we would not have in an audit.
Two ablations isolate what is new. The first drops the factorization and scores raw cosine consistency of activations across variants; it tests whether surface and logic must be removed before the shared identity signal becomes visible. The second removes the swap training and the gradient-reversal term; it tests whether a disentangled factorization matters or any reconstruction model would do.
Beating the field means, in Stage II and in Stage I after GRPO, an AUROC that clearly exceeds the best of these detectors on held-out items. We set the target at 75% AUROC, with true-positive rate at 5% false-positive rate reported alongside, and require it to hold across three seeds.
Why not just…?
Why not just filter training data with n-gram overlap? That approach requires the training corpus, which audits of released models usually lack. It also fails on reworded leaks: Rethinking Benchmark and Contamination shows that Llama-2-13B trained on reworded MMLU rises from about 55 to about 86, undetected by n-gram filters. Our rephrased-contamination arm tests exactly this case.
Why not just use Min-K% or LiRA? The Fragility paper shows that these detectors collapse to near chance in exactly the regimes we target, and the paper proves the collapse follows from PPO-style clipping rather than from poor tuning. We rerun them as baselines on identical checkpoints to confirm.
Why not just check whether the model gives the old answer on logic-edited variants? This is the most serious threat, because it uses the same variants with no internals. We run it as a behavioral baseline: the persistence rate of the original answer, plus the gap in accuracy between rewordings and logic edits. If it matches our AUROC, the latent machinery is unnecessary and we will say so. Our method should win in two situations. One is items where the memory pressure is subthreshold, meaning the model answers the edited problem correctly while still carrying the identity trace. The other is chain-of-thought contamination, where the model has absorbed a method and recomputes answers. We report results separately on items where behavioral persistence is zero, to see whether the hidden-state signal still detects them.
Why not just train a supervised probe on activations? Membership labels for the audited model are exactly what an auditor lacks. A probe trained on one contaminated model may also learn that model's particular artifacts. The transfer-probe baseline measures this directly. If it transfers well, it is a cheaper competitor, and we would adopt it.
Why not just evaluate on dynamic benchmarks like DyCodeEval? Dynamic benchmarks estimate how inflated a score is overall. They do not say which items leaked, and they depend on strong proprietary rewriters. Item-level detection is complementary, so we compare against them only for aggregate-inflation estimates.
Decisive Checks
The first check is whether $S$ separates contaminated from clean items in Stage II and after GRPO in Stage I, where output detectors sit at 50–58%. The second is a specificity control in the near-neighbor arm. Unseen items close to contaminated ones must not score high; otherwise $S$ detects familiarity with a skill rather than item identity. Chain-of-thought contamination generalizes to similar questions, so this control determines whether the claim holds at all. The third is dose response: $S$ should rise with the number of contamination epochs and fall only slowly with additional clean GRPO epochs. The fourth is the factorization ablation. If raw cross-variant consistency performs as well, the disentanglement component adds nothing, and the contribution reduces to the variant-probing protocol. The method needs model weights, so black-box API models are out of scope. That limitation is shared with all of the internal-probe baselines.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization
- Generalization or Memorization: Dynamic Decoding for Mode Steering
- Unsupervised Shape and Pose Disentanglement for 3D Meshes
- Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models