Answers That Arrive Too Early: Detecting Benchmark Contamination by When an Answer Forms Across Layers

A deep network builds its answer in stages. Early layers handle local, surface features, and later layers combine them into more abstract conclusions. A problem that needs several dependent reasoning steps should therefore need many layers, and often more than one forward pass, before its answer exists anywhere inside the model. This note proposes a contamination test based on that expectation. Before a reasoning model writes a single token of its chain of thought, we read its intermediate layers. If the correct final answer to a multi-step problem can already be decoded at a middle layer, the model had too little computation to have derived it. The most likely explanation is that it retrieved a stored answer. The test needs one forward pass over the question and no access to training data. It also works even when reinforcement learning has hidden contamination from detectors that look only at the model's output.

The timing matters because the cheap detectors have stopped working. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that ten output-level detectors drop to near-random accuracy after clean reinforcement learning or after contamination with full reasoning traces. Their area under the ROC curve (AUROC, the probability that a contaminated item scores above a clean one) falls from 89.1% to about 50%. The internal-representation detector LaRA survives reinforcement learning, but it needs several perturbed forward passes per item and a verified clean reference set drawn from a matching distribution. We replace that reference set with a principled expectation of how deep an answer should form, and we calibrate that expectation on procedurally generated problems that cannot be contaminated.

Core Claim

Some notation first. A lens maps each layer's hidden state into vocabulary space so that it can be read as next-token probabilities. The logit lens reuses the model's own output head for this. The tuned lens learns a small linear map per layer. For a question $x$ with gold answer $a$, let $p_\ell(a \mid x)$ be the probability the lens at layer $\ell$ assigns to $a$. This is read at the position immediately after the question and a fixed answer cue, before any reasoning tokens are generated. For multi-token answers we teacher-force the answer tokens and sum their per-token log-probabilities. The answer formation layer is

$$\ell^*(x) = \min\{\ell : \log p_\ell(a \mid x) - \log p_\ell(a' \mid x) > \delta \ \text{for all distractor answers } a'\},$$

where $\delta$ is a margin set on held-out clean data. We take $\ell^*(x) = \infty$ when no layer meets the condition. The contamination score is the peak pre-reasoning margin over layers, compared against what a clean item of the same compositional depth produces.

The claim has three parts. First, for clean problems, $\ell^*$ should move to deeper layers, and then to $\infty$, as the number of dependent steps grows. This is the hierarchical-composition expectation applied to a transformer. Second, contaminated items break this curve: they form early even at high depth. Third, this signal survives reinforcement learning, because clipping in PPO and GRPO (two common reinforcement learning algorithms for language models) acts on output log-probabilities rather than on mid-layer retrieval.

Spurious Rewards Paradox motivates the second and third parts. It finds that leaked MATH-500 answers in Qwen2.5-Math are latently present around layers 18–20, the band it calls "functional anchors," and that later layers only reformat them. That paper studies one model family and uses interventions to explain a mechanism. We turn the observation into a detector that does not depend on any particular architecture, and we test whether the band is a general property of transformers.

Grounding in the Two Directions

The hierarchical-learning literature supplies the expectation that answers need depth. Studies in vision and speech show that stacked layers outperform equally sized shallow ones and that removing a single layer hurts accuracy (ImageNet Classification with Deep Convolutional Neural Networks, Return of the Devil in the Details). Others show that invariant, global features appear only deep in the network (Aircraft Detection by Deep Convolutional Neural Networks). These are convolutional results on perception tasks. We treat them as a hypothesis to test in transformers, not as evidence for it.

DyVal supplies the testbed. It generates reasoning problems as directed acyclic graphs (DAGs, graphs where each node depends on earlier nodes) with controlled depth from $D_1$ to $D_4$. It reports that accuracy falls much faster with graph depth than with graph width. It does not look inside the model, and we do.

Two papers constrain the design. Can Neural Network Memorization Be Localized? finds that memorization in vision models is spread across all layers rather than confined to the last ones. LaRA likewise finds contamination to be a distributed geometric effect. Our score therefore does not assume a single anchor band. It takes the earliest layer at which the answer becomes decodable, which is well defined whether retrieval is spread out or concentrated. Measuring how sharply $\ell^*$ clusters across items and model families directly addresses the open disagreement between LaRA's distributed picture and the anchor-adapter circuit's localized one.

ROME shows that memorized and non-memorized facts separate in hidden-state space. It studies single-entity recall without layer-by-layer trajectories. We extend the question to multi-step benchmark items and to the depth at which they separate.

Research Program

The first stage builds a clean depth curve. We generate fresh DyVal instances across all seven tasks at $D_1$ through $D_4$ and compute $\ell^*$ profiles with both the logit lens and the tuned lens. We do this for open-weight families: Qwen2.5-Math, Llama-based DeepSeek-R1-Distill-Llama-8B, and at least one Gemma or Mistral model. The output is, for each model, the distribution of peak pre-reasoning margin as a function of graph depth. This stage tests the transfer of the hierarchical-composition expectation to transformers. If $\ell^*$ does not increase with DAG depth on clean items, the core premise fails, and we report that as the result.

The second stage plants contamination with known ground truth. We follow the two pipelines from the Fragility paper. In Stage I, a subset of items is included in supervised fine-tuning, followed by clean GRPO. In Stage II, full chain-of-thought traces are used for supervised fine-tuning of DeepSeek-R1-Distill-Llama-8B. We apply both pipelines to two pools. The first pool is DyVal instances, which gives exact membership labels at every depth. The second pool is competition math, namely AIME and MATH-500 items, with post-cutoff AIME 2026 and LiveMathBench as non-members. We also evaluate on the existing RL-MIA Math and PRIME-RL member/non-member split used by LaRA, so that we can compare against LaRA directly on its own setting.

The third stage scores natural benchmarks without planted labels. On models whose contamination status is plausibly known, we compare the score distributions of MATH-500 and LiveMathBench, following the setup in Spurious Rewards Paradox. For competition items we do not know the compositional depth. The calibration therefore uses a weaker but cheap reference: post-cutoff problems of similar difficulty. This reference is defined by date rather than by distribution matching, which is a looser requirement than LaRA's.

Baselines

The naive reference is forced direct answering. We prompt the model to give its final answer with no reasoning and score each item by the final-layer log-probability margin of the gold answer. This is our detector restricted to the last layer, so it serves both as the "do nothing clever" point and as the ablation that isolates the proposal's key ingredient, which is reading intermediate layers. It uses the same prompts, the same answer cue, and a single forward pass. If our method does not beat it, the layer-wise idea adds nothing.

The output-level membership detectors are Min-K% token probability, loss, and LiRA, as implemented in the Fragility paper. We run them on identical contaminated checkpoints. The Fragility paper reports that these detectors fall to roughly 50–58% AUROC after clean reinforcement learning, from 89.1% beforehand. Our method beats them if it holds AUROC well above that range on the same post-GRPO and Stage II checkpoints.

The strongest existing competitor is LaRA, because it is the only detector shown to survive reinforcement learning using internal representations. On its RL-MIA and PRIME-RL setting, LaRA reports ROC-AUC 0.79 and a 31–38% true-positive rate at 5% false-positive rate, in combination with Self-Critique. We run LaRA with its three perturbation types and its clean median-absolute-deviation reference, on the same samples. We report two comparisons: one at equal forward-pass budget, where LaRA receives its full perturbation set and we report the cost ratio, and one where our single-pass score is allowed to use the same number of passes by averaging over paraphrases. Beating LaRA means exceeding its true-positive rate at 5% false-positive rate on its own split, or matching it at a fraction of the compute without its distribution-matched clean reference.

A probe-based audit in the style of Spurious Rewards Paradox is a fourth comparison. It uses a fixed layer-20 probe on Qwen and is tested for transfer to other families. This checks whether a single anchor layer suffices or whether the full depth profile is needed.

Two further ablations separate the ingredients. The first swaps the logit lens for the tuned lens. The second removes depth calibration, keeping the raw peak margin while dropping the comparison to the DyVal curve.

Why not just…?

Why not just ask the model to answer without reasoning and flag correct direct answers? This is the most serious threat, and it is exactly the forced-direct-answer baseline. The case against it is that reasoning-trained models learn to emit answers only after thinking. Spurious Rewards Paradox describes late "adapter" layers that reformat a retrieved signal, and the Fragility paper shows reinforcement learning compressing output log-probability gaps. A latent mid-layer answer can therefore be suppressed at the output. Whether that suppression happens enough to matter is an empirical question. If direct answering matches our score on the post-GRPO and Stage II checkpoints, we will report the cheaper method as sufficient and narrow our contribution to the depth-curve analysis.

Why not just use LaRA? LaRA already reads internal layers. It needs multiple perturbed passes and a clean profile from a matching distribution, both of which its authors list as limitations. It was also not tested on PPO/GRPO-cloaked models or on chain-of-thought contamination. The head-to-head comparison on LaRA's own split, plus the Fragility pipelines, settles whether our score is better, cheaper, or complementary. If the two are complementary, we will report the combination.

Why not just replace static benchmarks with dynamic ones like DyVal? Dynamic generation does not audit models that have already been deployed, and it is currently limited to formal tasks. We use DyVal for calibration because generated items are guaranteed clean. That is the reason it complements a white-box audit rather than replacing one.

Why not just watermark benchmarks? Detecting Benchmark Contamination Through Watermarking reaches $p = 10^{-3}$ for a 5% gain on ARC-Easy. It cannot be applied retroactively to MMLU, MATH, or AIME, and its authors note that rephrasing can damage formal math items. Those legacy benchmarks are exactly our targets.

Decisive Checks

The premise check comes first. On clean DyVal items, $\ell^*$ must increase with depth from $D_1$ to $D_4$ in every model family. If it does not, the composition hypothesis fails in transformers.

The main check uses the planted-contamination checkpoints. There we report AUROC and true-positive rate at 5% false-positive rate before and after GRPO, against all baselines. Success means that output detectors collapse toward 50% while the layer-wise score does not. It also means that the layer-wise score beats forced direct answering by a clear margin, which the planted DyVal labels make measurable at each depth.

On the localization question, we report the spread of $\ell^*$ across items and model families. A tight band that holds across families supports the anchor-adapter account. A broad spread supports LaRA's distributed account. Either outcome is informative.

The final check is a falsification test. Contaminated items should show a flat formation curve across DyVal depth, while clean items show a rising one. If the curves have the same shape, the signal is tracking problem difficulty rather than memorization.