Large language models are often trained, by accident or not, on the test sets they are later scored on. Contamination detectors catch this by noticing that a model is unusually confident on items it has seen. Reinforcement-learning post-training (the reward-driven fine-tuning now standard for reasoning models) breaks those detectors. After a single epoch of GRPO on clean data, a model that was contaminated during supervised fine-tuning looks clean to loss- and likelihood-based tests, while its inflated scores remain (On The Fragility of Benchmark Contamination Detection in Reasoning Models). A separate line of interpretability work points to where the evidence might still be. Jet expansions rewrite a transformer's residual computation as an explicit sum of polynomial input-to-output paths through subsets of layers plus a remainder term. Applied to aligned models, they show that RLHF masks harmful associations by adding "refusal mass" in the final layers and leaves the underlying paths unchanged (Decomposing LLM Computation with Jets).
This proposal puts those two findings together. If RL mostly edits the last few blocks, memorized benchmark answers should still be readable from the part of the network that RL left alone. We therefore propose to read a model's confidence on benchmark answers at every depth, using truncated jet expansions. These are polynomial approximations, borrowed from direct symbolic regression, that route the residual stream to the output while replacing the later blocks with their low-order Taylor terms. The detector then asks whether an item's answer is anomalously likely at some intermediate depth, even when the final output shows no anomaly.
Core Claim
The claim is that post-RL contamination is hidden at the output but visible at intermediate depths. We also claim that first-order jet terms recover more of it than the zero-order case, which is the familiar "logit lens" of decoding intermediate activations directly. Concretely, let $p_\ell(y \mid x)$ be the probability of reference answer $y$ given benchmark prompt $x$ under the jet expansion truncated after block $\ell$: blocks $1..\ell$ are kept exactly and blocks $\ell+1..L$ enter only through their order-$k$ jet terms. Define the per-item score as
$$ s(x) = \max_{\ell} \frac{\log p_\ell(y \mid x) - \mu_\ell}{\sigma_\ell}, $$
where $\mu_\ell$ and $\sigma_\ell$ are the mean and standard deviation of the same quantity on a reference set of items known to be unseen. The reference set could be post-cutoff problems or held-out generated instances. If RL clipping compresses the likelihood gap mainly in late blocks, as the fragility analysis and the jet RLHF result jointly suggest, then $s$ should separate seen from unseen items where the final-layer loss cannot.
What is new here is not the use of internal states for contamination, which RADAR already does. It is an unsupervised detector with a stated mechanism, tested specifically against the RL cloaking effect. It also comes with an ablation that asks whether the polynomial machinery earns its cost over plain logit-lens decoding.
Relation to Prior Work
Two papers motivate the gap. The fragility study shows that PPO-style importance clipping shrinks the log-probability gap between seen and unseen items. Its ten detectors fall to 50–58% AUROC once reasoning models are contaminated with chain-of-thought data. Its authors propose no replacement. Detecting Data Contamination from Reinforcement Learning Post-training introduces Self-Critique, a black-box probe that compares token-entropy patterns between an answer and a prompted alternative, and the RL-MIA benchmark. Self-Critique works only where rewards are verifiable and says nothing about where in the network memorization lives. We keep its benchmark and treat it as the strongest competitor.
The method comes from direct symbolic regression. HOPE expands trained networks into local Taylor polynomials via a generalized chain rule, but only on small MLPs and CNNs. Jet Expansions of Residual Computation and its LLM follow-up extend this to transformer residual streams, report logit fidelity often above 0.9 cosine similarity, and show that the logit lens is the zero-order jet. They have only audited short associations such as bigrams and toxicity, never test-set memorization.
The jet papers also constrain the design. The number of jet paths grows super-exponentially with depth, and second and higher orders are prohibitive at LLM widths. We therefore use only the $L$ "exit" truncations, one per depth, at orders zero and one. We make no attempt to enumerate paths.
RADAR is the closest existing detector. It trains classifiers on 37 surface and mechanistic features, but uses 30 training examples and was never tested after RL. Performance-based methods (ConStat), output-sampling methods (CDD/TED), and in-context methods (CoDeC) all read the final output only. The fragility result suggests exactly that output is what RL rewrites.
Research Program
The build is modest. For each open-weight model, we implement exit truncations at every block. At order zero, the residual stream after block $\ell$ is passed through the final norm and unembedding. At order one, the skipped blocks are replaced by their linearization around the residual state actually reached on the input, computed with one Jacobian-vector product per skipped block. This matches the locality caveat from HOPE: the expansion point is the item's own trajectory, so we never extrapolate far from it. We score teacher-forced reference answers, and for chain-of-thought benchmarks also reference solutions, under each truncation.
We first check fidelity. At each depth, the order-one truncation must match or exceed logit-lens agreement with the full model's next-token distribution on clean text. Otherwise the first-order comparison is meaningless.
The analysis has three parts. The first is a depth profile of the calibrated gap between seen and unseen items, taken before and after RL, to test whether RL erases the gap from the top down. The second is a per-block comparison of weights between SFT and RL checkpoints, where we have both, to test whether the depth at which the signal survives lines up with where RL changed parameters least. The third is a real-world audit on public open-weight reasoning models, using post-cutoff items as the unseen reference.
The approach has a hard limit. It needs weights, so it cannot audit API-only models.
Baselines
The naive reference point is final-layer loss and Min-K% on the same items. These are the scores the fragility paper shows collapsing toward chance. Every method is scored as item-level AUROC on identical seen/unseen splits. These baselines anchor the claim that anything is being recovered at all.
The strongest existing method is Self-Critique (Detecting Data Contamination from Reinforcement Learning Post-training). On RL-MIA it reports an average AUC of 0.70 on Qwen2.5-7B-Instruct and 0.64 on DeepSeek-Math-7B-Instruct, against 0.42–0.53 for perplexity and Min-K%. We run its published protocol on the same two models and the same AIME, Knights & Knaves, and SAT subsets. Its sampling budget is matched in wall-clock compute to our $L$ forward-plus-JVP passes. Beating it means a higher average AUC on both models, with confidence intervals from bootstrap over items that do not overlap 0.70 and 0.64 respectively. For the fragility protocol, SFT contamination followed by GRPO on clean data, we did not find a reported result on Lacuna for Self-Critique. There the bar is the reported 50–58% AUROC of the ten standard detectors.
CoDeC and CDD are run as gray-box baselines with their default settings. On RL-MIA, the file has no reported result on Lacuna for CoDeC, and none for CDD.
Two ablations isolate the key ingredient. The first uses the logit lens alone, meaning the same depth-max score at order zero. This tests whether the polynomial terms matter or whether depth alone does the work. The second is a supervised linear probe on residual activations at each depth, trained on one model's labeled seen/unseen split and tested on another. It measures how much of our unsupervised score a cheap learned detector could match.
An oracle is also reported. It is the loss difference between the pre-RL SFT checkpoint and a clean reference model, which gives an upper bound on what survives in the weights.
Why not just…?
Why not just use the logit lens, since it is the zero-order jet anyway? This is the most serious threat, and it may win. If most of the surviving signal sits in intermediate residual states, decoding them directly may be enough, and first-order terms add cost for little gain. The order-zero ablation settles this directly. If order one does not beat order zero by a clear AUROC margin on RL-MIA and on the fragility protocol, we will report the finding as "depth-resolved logit-lens detection." We will not claim the polynomial machinery helps. One reason to expect order one to help: intermediate states are not in the output basis, so the logit lens is known to be poorly calibrated at early depths. Linearizing the skipped blocks corrects that without training.
Why not just train a probe on activations, as RADAR does? Probes need labeled contaminated examples from a comparable model, and auditors rarely have them. RADAR's own accuracy drops to 76.7% on ambiguous cases with 30 training examples. Our cross-model probe baseline measures the gap honestly. If a probe transfers across models as well as the unsupervised score, the practical case for jets narrows to not needing labels.
Why not just compare against the pre-RL checkpoint, or use performance-gap tests like ConStat? Checkpoint comparison is the oracle, and auditors of released models almost never have the SFT checkpoint. ConStat detects inflated scores for a whole model and benchmark rather than identifying which items leaked. It also depends on a fleet of reference models. We view it as complementary and report it at the dataset level where feasible.
Decisive Checks
Three results would decide the proposal. First, in the fragility-style setup, the final-layer loss gap should vanish after GRPO while the calibrated gap at some intermediate depth persists. In a clean sibling model trained identically without contamination, no depth should show a gap. That second condition is the false-positive control. Without it, depth selection by maximum alone could manufacture signal. Second, on RL-MIA the depth-max score should exceed Self-Critique's 0.70 and 0.64 AUC. If contamination during RL itself leaves no intermediate-depth trace, that is a clean negative result: it would mean RL-phase memorization, unlike SFT-phase memorization, is written into the whole network. Third, order one should beat order zero. The Stage II chain-of-thought case is likely to be hardest, since the fragility authors argue those models generalize reasoning rather than memorize text. If our score stays near chance there too, that supports their reading that such contamination is not memorization in any internal sense. We would report it as such.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Decomposing LLM Computation with Jets
- RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
- Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
- HOPE: High-Order Polynomial Expansion of Black-Box Neural Networks
- Jet Expansions of Residual Computation
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- Detecting Data Contamination in LLMs via In-Context Learning