Many classifiers now "explain, then predict." A language model first writes a short natural-language justification for an answer, and the answer is then read off conditioned on that justification. The hope is that the model decides because of the explanation. This note argues that benchmark contamination, meaning test questions leaked into the model's training data, breaks that hope in a specific and measurable way. When a model has memorized a test item, the final answer is routed straight from the memorized question text to the memorized label by a handful of late-layer attention heads. The explanation the model just wrote is skipped. This is the same mechanism that backdoored ("Trojaned") Transformers use, where a few heads lock onto a trigger token and drown out everything else. Here, though, nobody planted the trigger. The leaked item itself plays that role.
If this holds, two practical things follow. First, a cheap behavioral test can flag contaminated items. We hand the model a counterfactual explanation that argues for a different answer and check whether its prediction follows that explanation. A contaminated model will ignore the counterfactual. Second, a mechanistic repair becomes possible: once the "bypass heads" are found, pruning them should reveal how the model would score if it actually used its reasoning. The hardest part is a confound. A model that simply knows the answer from legitimate training will also ignore a wrong explanation. The central experiment is built to tell memorizing the item apart from knowing the fact.
Core Claim
In an explain-then-predict pipeline, contamination shows up as rationale bypass, a drop in how much the answer depends on the generated explanation. The drop is carried by a small set of deep-layer heads whose answer-time attention concentrates on item-specific surface tokens of the question. Examples of such tokens are the exact wording of the distractor choices or rare phrasing. The heads attend there instead of to the explanation tokens.
We define rationale dependence for an item $x$ as $D(x) = \Pr[\hat y(x, r') = a']$. Here $r'$ is a fluent explanation the same model writes when asked to justify a wrong answer $a'$, and $\hat y(x, r')$ is the pipeline's prediction when $r'$ is substituted for its own explanation. The claim makes three predictions. Contaminated items have lower $D$ than clean items of matched difficulty. Items that the model answers correctly from injected knowledge of the underlying fact, but not of the item, keep intermediate $D$ and show a different attention pattern. Ablating the localized heads raises $D$ on contaminated items and pulls their accuracy toward the accuracy of an uncontaminated twin model.
How This Relates to Prior Work
The contamination literature supplies both the motivation and the main competitors.
Detecting Pretraining Data from Large Language Models detects memorized text by averaging the lowest-probability tokens (MIN-K% PROB). Its authors note that it degrades under paraphrase, and it says nothing about whether a pipeline's decision is driven by memorization.
RADAR is the closest work. It shows that recall produces focused attention and specialized heads, while reasoning produces distributed attention, and it trains a classifier on 37 such features. RADAR studies direct question answering on 30 training and 100 test examples, with no ground-truth contamination control. It never asks whether an explanation step disperses or hides the recall signature. We differ in three ways. Our object is the answer step's dependence on the explanation, not attention statistics of a direct answer. We use controlled injection so contamination is known rather than labeled by judgment. We intervene causally rather than only classify.
DyVal and Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? show large score drops on dynamically generated or post-cutoff data. Both stay at the input-output level, and DyVal's procedure is limited to formal tasks.
The attention-hijacking literature supplies the method. A Study of the Attention Abnormality in Trojaned BERTs and Attention Hijacking in Trojan Transformers locate "hijacking heads" concentrated in the final three layers. They report that pruning these heads restores up to 30% accuracy on poisoned inputs. Backdoor Attacks in Token Selection of Attention Mechanism proves that gradient descent drives a poisoned token's attention weight to about 1. Memorization in Attention-only Transformers shows attention alone can act as a lookup table. Together these make it plausible that memorized items are routed through attention, but all of them study planted triggers or synthetic associations. We borrow their head-localization and pruning tools and point them at an unplanted, distributed "trigger": the leaked item.
Two papers constrain the design. Chain-of-Thought Hijacking shows that long reasoning chains pull attention onto the model's own generated tokens and dilute prompt-level signals. That is the opposite failure, and it means we cannot assume explanations are ignored in general. We must measure bypass per item, and we should expect clean items to depend heavily on explanations. Explain Yourself! introduced the explain-then-predict pipeline on CommonsenseQA (CAGE, 64.7% test accuracy versus 54.8% prior). It flagged dataset artifacts but never checked memorization. Its finding that human explanations push accuracy to about 90% shows that explanations can carry the decision, which is what makes bypass meaningful when it occurs.
Research Program
Controlled contamination. We take open decoder models whose pretraining corpus is public. Models trained on The Pile, as used in Snoopy, let us check pre-existing exposure. We continue pretraining twin copies under four conditions that differ only in what is injected:
- Clean: nothing injected.
- Item contamination: a random half of CommonsenseQA dev items, inserted verbatim with gold answers at one, four, or sixteen repetitions.
- Paraphrased item contamination: the same items reworded, keeping choices and answers.
- Knowledge injection: the underlying ConceptNet-style fact behind each selected item (e.g., "a bassoon is found in an orchestra"), written in varied non-item sentences.
The Winograd Schema data from Back to Square One serve as a second task, because twin sentences give natural matched controls. Each twin model is fine-tuned into an explain-then-predict classifier using CoS-E explanations, following CAGE.
Measurement. For every test item we compute three quantities:
- $D(x)$, using model-generated counterfactual explanations. We filter these with a separate judge model so they are fluent and actually argue for $a'$.
- The answer step's attention split between question tokens and explanation tokens, per head.
- A bypass score per head, defined as the drop in $D$ when that head's output is mean-ablated, following the head-ablation procedures of the Trojan work.
Heads whose ablation most raises $D$ on contaminated items, measured on a held-out calibration split, form the bypass set. We check whether these heads overlap with heads found by applying AttenTD-style analysis to a deliberately Trojaned SST-2 explain-then-predict model. That comparison is the bridge test of whether contamination and backdoors share circuitry.
Natural contamination. We then apply the same instruments to models where contamination is suspected but not controlled. We compare older causal benchmarks (COPA, e-CARE) against post-cutoff CausalProbe-2024, and WIKIMIA member versus non-member passages turned into cloze questions.
Baselines
The naive reference point is reporting explain-then-predict accuracy as is, and comparing accuracy with and without the explanation step. If contamination inflated both equally, this gap would carry no signal. We report it so readers can see that standard pipeline numbers hide the effect.
The strongest surface-level competitor is MIN-K% PROB from Detecting Pretraining Data from Large Language Models. It is the most widely used reference-free detector and needs only log-probabilities. The file reports it as a relative gain rather than an absolute score: a 7.4% AUC improvement over the prior best on WIKIMIA and a 12.2% improvement in detecting intentional benchmark leakage. We run it on the same twin models and the same items with $k$ tuned on the same calibration split. We score by area under the curve (AUC) for separating injected from non-injected items. Beating it means higher AUC on verbatim contamination and, more importantly, a clear margin on the paraphrased condition, where its authors report degradation.
The strongest mechanistic competitor is RADAR, which reports 93.0% overall accuracy on its 100-example test set and 76.7% on ambiguous cases. We reimplement its feature set on the answer step of our pipeline and train its classifier on the same calibration items we use. The comparison is matched on labeled data, not on RADAR's original 30 examples. Beating it means higher AUC on our controlled split, and specifically separating item contamination from knowledge injection, which RADAR's recall-versus-reasoning features are not designed to do.
Three ablations isolate our ingredients:
- The behavioral test alone: $D(x)$ with no attention analysis. This tells us whether the mechanistic layer adds anything for detection.
- Attention ratio without localization: the raw question-versus-explanation attention ratio averaged over all heads. This tests whether localizing the bypass heads matters.
- Untargeted pruning: pruning an equal number of random heads, or heads chosen by activation magnitude in the style of standard pruning. This tests whether the repair depends on finding the right heads.
For repair, the competitor is attention-spike regularization from Obliviate. It cut attack success by 83.6% at a 0.78% clean-accuracy cost. We apply it during the classifier fine-tune with the same parameter budget.
The repair target is the clean twin's accuracy on the injected items. The metric is the absolute error between the repaired contaminated model's score and that target. Success means a smaller error than Obliviate and random pruning, with clean-item accuracy within one point of unrepaired.
Why not just…?
Why not just use MIN-K% PROB? It answers a different question: whether the text was seen, not whether the decision was driven by having seen it. It is also known to weaken under paraphrase. Our paraphrased-contamination condition tests directly whether a decision-level signal survives where likelihood signals fade. If MIN-K% matches us on paraphrase, the detection half of this proposal loses its point, and we would say so. Its blind spot for repair would remain.
Why not just prompt for chain-of-thought and trust the explanation? Writing an explanation does not guarantee the answer uses it, which is the faithfulness gap noted for explain-then-predict systems. On post-cutoff CausalProbe-H, chain-of-thought did not prevent drops to below 70% for Claude 3 Opus and about 56% for LLaMA 2, per Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?. Our with-versus-without-explanation baseline measures this directly on known contaminated items.
Why not just run the counterfactual-explanation test and skip attention entirely? This is a real threat. For detection alone, $D(x)$ might do everything, and it works through a black-box interface. We treat it as an ablation, not a strawman. The attention analysis must earn its place in three places. It must separate item memorization from fact knowledge, since both lower $D$. It must enable repair, which the behavioral test cannot do. It must explain why bypass happens, through the overlap test with Trojan heads. If the head-level features fail to beat $D$ on the knowledge-versus-item split and repair does not beat random pruning, we will report the behavioral test as the contribution and drop the mechanistic claims.
Why not just use dynamic benchmarks? DyVal works for formal tasks, but generating fresh commonsense questions procedurally remains unsolved. Benchmarks like CommonsenseQA and MMLU stay in use, so their scores need auditing in place.
Why not just prune generically? Backdoor Attacks Can Transfer to All found that hijacked pathways run through highly active structural routes and survive pruning of inactive neurons. Only targeted removal of hijacking heads restored poisoned accuracy in the Trojaned-BERT study. Our random-pruning and magnitude-pruning ablations test whether that holds for contamination.
Decisive Checks
The first check is the knowledge-versus-item split. On items whose gold answer the model gets right in both the item-contaminated and knowledge-injected twins, does the bypass-head signal separate the two conditions with AUC well above what $D$ alone achieves? If it does not, contamination and genuine knowledge are mechanistically indistinguishable at this level, and the proposal's main mechanistic claim fails.
The second check is the dose-response curve. Bypass should grow with injection repetitions (one, four, sixteen) and should not appear in the clean twin. A flat curve would mean $D$ reflects item difficulty, not memorization.
The third check is repair fidelity. After ablating the bypass heads, the contaminated twin's accuracy on injected items should land within a few points of the clean twin's accuracy on those same items, while non-injected accuracy stays put.
The fourth check is the shared-circuit test. If bypass heads overlap with the Trojan hijacking heads well beyond the overlap expected for random head sets of the same size, contamination is acting as an unplanted backdoor. If they do not overlap, the analogy is only descriptive, and we will report it that way.
References
- Detecting Pretraining Data from Large Language Models
- RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- A Study of the Attention Abnormality in Trojaned BERTs
- Attention Hijacking in Trojan Transformers
- Backdoor Attacks in Token Selection of Attention Mechanism
- Memorization in Attention-only Transformers
- Chain-of-Thought Hijacking
- Explain Yourself! Leveraging Language Models for Commonsense Reasoning
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm
- Backdoor Pre-trained Models Can Transfer to All