When a language model has seen a test question and its answer during training, it can produce the right answer by recall and then write a step-by-step explanation that looks like the source of that answer but is not. This proposal tests that claim directly. We build reasoning problems from known computation graphs, so every intermediate value has a ground truth. We deliberately leak some of these problems into a model's training data. We then check, one step at a time, whether the model's final answer actually follows from the intermediate values it wrote down. If contamination makes models rationalize after the fact, the answer should stop tracking the trace on leaked items and keep tracking it on their unleaked twins.
Two things would follow if this holds. First, contamination becomes detectable for a single item, from the model's own output, without a clean reference model. Today's detectors only compare benchmark-level accuracy against rephrased or reference distributions. Second, there is a principled fix. Forcing the model to compute each step in isolation, seeing only that step's inputs, removes the cue that lets it recall the answer. The accuracy it loses under this forced decomposition then measures how much of its score was memorized.
Core Claim
Our claim is that benchmark contamination does more than inflate accuracy. It changes how answers are produced, shifting the model from "trace causes answer" toward "question causes both trace and answer." This is the "common cause" structure that How Likely Do LLMs with CoT Mimic Human Reasoning? identified in general chain-of-thought (CoT) generation. That paper found GPT-4 reached correct 6-digit sums despite wrong steps 74.4% of the time, but it did not ask what causes this pattern. We propose contamination as a controllable cause. We measure it with an audit that needs no judge model and no sampled counterfactuals, because the problem's graph already tells us what the answer should be given any set of stated intermediate values.
Concretely, let $G$ be the directed acyclic graph that generated a problem. Let $a_1,\dots,a_k$ be the intermediate values the model asserts in its trace, extracted as atomic claims of the form "node $v$ equals $x$." Let $f_G(a_1,\dots,a_k)$ be the answer obtained by composing those stated values through $G$'s operators. Let $\hat y$ be the model's final answer. We define two per-item quantities. The first is decoupling, $\mathbb{1}[\hat y \neq f_G(a_1,\dots,a_k)]$: the answer disagrees with the model's own stated steps. The second is edit-following. We overwrite one asserted value $a_j$ in the trace with $a_j'$, let the model continue, and record
$$E_j = \mathbb{1}\left[\hat y' = f_G(a_1,\dots,a_j',\dots,a_k)\right],$$
where $\hat y'$ is the answer after the edit. A model that reasons through its trace follows the graph-predicted answer. A model recalling a memorized answer keeps $\hat y' = \hat y$.
How This Relates to Prior Work
Three lines of work motivate the question but stop short of it. Contamination detectors measure outputs only. ConStat flags a model when its accuracy on the original benchmark exceeds what a reference difficulty curve predicts from rephrased variants. DCR scores leakage at four levels, from semantic exposure to label exposure, and discounts accuracy accordingly. Neither looks inside the reasoning trace, and ConStat needs clean reference models, which it concedes may not exist for ubiquitous benchmarks. DyVal generates fresh problems from graphs to sidestep contamination. It shows phi-1.5 and WizardMath-13B scoring near zero and a 13.6% GPT-4 drop when description order is flipped, but it scores only final answers. We reuse DyVal's generator as our method substrate, using the graph as an oracle for step-level faithfulness rather than only as a source of clean items.
Faithfulness methods supply our intervention technique. Measuring Faithfulness in Chain-of-Thought Reasoning introduced early answering and error injection. We keep error injection but replace its "did the answer change?" test with "did the answer change to the graph-predicted value?", which separates ignoring an edit from being derailed by it. FRIT swaps steps for style-matched distractors and trains on the results. It pays heavy inference cost and tests only on static benchmarks, where contamination is uncontrolled. FaithCoT-Bench labels instance-level unfaithfulness with expert step evidence. It reports that counterfactual edits break on wordy, redundant text and that logit-based detectors often fall below 50 F1. That finding constrains our design. We deliberately work in a domain where each atomic step is a single non-redundant node value, so interventions are clean.
Atomic decomposition supplies the audit format. Ask, Don't Judge shows that atomic binary checks track human judgment better than holistic judges. We apply the same idea to reasoning steps, where each atom is checkable exactly against the graph rather than by an LLM. Modular reasoning supplies the mitigation. FaiRR restricts its generator's context to selected rules and facts and reaches near 1.0 precision on intermediate inferences. FaithformBench is a cautionary counterpart: decomposition into formal code can still "silently correct" invalid input, with false-positive rates often above 40% on harder sets. That warns us that modularity enforces faithfulness only if each module truly cannot see the global answer cue. We test exactly that condition.
Research Program
The first component is a twin-controlled contamination suite. Using DyVal-style generation, we build arithmetic, linear-equation, and Boolean-logic graphs at depths D1–D4. Each graph is instantiated twice with different leaf values and surface entities. This produces "twins" that share topology and difficulty but not answers. One twin of each pair is injected into fine-tuning data for open 7–8B models in the Mistral-7B and Qwen3-8B families, which the cited papers already study. We use three exposure regimes, mirroring DCR's levels: question plus answer only (label leakage), question plus a correct CoT plus answer (full solution leakage), and a paraphrased question plus answer (semantic leakage that defeats n-gram filters). Exposure counts vary from one to many repetitions. Clean twins and a no-injection model serve as controls. Because contamination is assigned by us, every item has a true label, which no static benchmark offers.
The second component is the trace audit. A deterministic parser turns each CoT into atomic node-value claims, with an LLM fallback whose agreement with the parser we report. We then compute atom correctness, decoupling, and edit-following at every internal node.
The third component is a modular execution mode. The model evaluates one node at a time and sees only that node's operator and the values of its parent nodes, never the full question. The final answer is composed outside the model. The gap between monolithic and modular accuracy on leaked items, minus the same gap on clean twins, becomes a reference-free contamination estimate.
The fourth component transfers the audit to a static benchmark without a known graph. GSM8K solutions are themselves small arithmetic graphs, so we reconstruct the dependency graph from the model's own stated equations. Decoupling and edit-following are then measured on original GSM8K versus rephrased variants for models that ConStat reports as inflated.
Baselines
The naive reference point is raw accuracy with CoT on leaked versus clean twins. It shows that leakage inflates scores but cannot flag a single item without already knowing its twin. Every proposed signal must beat chance at separating leaked from clean items on its own.
The strongest existing contamination method is ConStat. In controlled experiments it detected 98.4% of sample-specific contamination against 71% for the best prior baseline, and it estimates a δ of about 8.3% inflation for Mistral-7b-v0.1 on GSM8K (ConStat). ConStat works at the benchmark level, so we compare in two ways. First, we compare at its own level: does a benchmark-averaged decoupling rate rank our injected models by exposure as well as ConStat's δ does, while using no reference models? Second, we compare at item level, where ConStat does not apply, against the baselines below. We give ConStat the same rephrased twins as its reference distribution and the same reference models we have access to. Beating it means matching its model-level detection without reference models, and providing item-level detection it cannot.
The strongest per-item faithfulness detector is the Faithful-Judge LLM judge from FaithCoT-Bench, at 65–77 F1. We run it on the same traces with the same judge-model budget we use for parsing fallback. Beating it means higher area under the ROC curve (AUROC) for leaked versus clean items, especially in the full-solution regime, where traces are fluent and correct. For likelihood-based membership detection (scoring how unsurprising the question is to the model), we found no reported result on Lacuna for instance-level contamination AUROC. We will run a min-probability token baseline ourselves on identical items. FaithCoT-Bench's finding that logit methods often score under 50 F1 for faithfulness is suggestive but concerns a different task.
Early answering from Measuring Faithfulness in Chain-of-Thought Reasoning means truncating the CoT and asking for the answer. It is calibrated per depth against clean twins, since shallow problems are solvable without CoT. Two ablations isolate our key ingredient. The first replaces graph-predicted edit-following with plain "answer changed" scoring. The second runs the modular mode with the full question visible to each node module. If either ablation matches the full method, the graph oracle or the context restriction, respectively, is not doing the work.
Why not just…?
Why not just use ConStat? For model-level auditing on benchmarks with good reference models, it may be enough, and its 98.4% controlled detection rate is a high bar. It cannot say which items are leaked, and it degrades when no clean reference exists, a limitation its authors acknowledge. Our reference-free comparison on injected models will show whether trace-level signals recover its rankings. If they do not, the item-level claim stands alone and we will say so.
Why not just measure question likelihood? For verbatim label leakage this is a real threat and may match or beat us, since memorized text is unsurprising. Our expectation is that it weakens under paraphrased leakage, where the evaluated question was never seen verbatim. Our expectation is also that it says nothing about whether the CoT is faithful. The paraphrase regime settles the detection comparison. If likelihood wins everywhere, we reframe the contribution as characterizing how contamination reshapes reasoning, not as a better detector.
Why not just use early answering? This is the closest simple competitor. If a model answers a depth-4 problem correctly with no CoT while its clean twin fails, that alone is suspicious. Early answering gives one bit per item, however, and cannot catch the case where a model genuinely uses part of its trace and recalls the rest. Edit-following localizes the decoupling to specific nodes. The depth-calibrated head-to-head will show whether that extra resolution improves detection. If it does not, early answering is the recommended practical tool, and our audit is useful mainly for diagnosis.
Why not just prompt the model to reason without assuming the answer? Measuring and Mitigating Post-Hoc Rationalization in RCG found that telling models to ignore a known answer increases subtle anchoring, and that structural decoupling was needed instead. We include such a prompt as a mitigation baseline next to modular execution, expecting it not to close the leaked–clean gap.
Decisive Checks
The central prediction holds if, under label-only and paraphrased leakage, decoupling rises and edit-following falls on leaked items relative to clean twins, increasing with exposure count and with depth. The most informative outcome is the full-solution regime. If traces there are atomically correct yet edit-following drops, contamination produces faithful-looking but non-causal reasoning, which an LLM judge should miss and our audit should catch.
The modular check succeeds if node-isolated execution brings leaked-item accuracy down to clean-twin accuracy while leaving clean-twin accuracy roughly unchanged. Its full-context ablation should not do this. The transfer check succeeds if Mistral-7B shows higher decoupling on original GSM8K than on rephrased variants, in the direction of ConStat's δ estimate.
A null result is also informative. If leaked items show inflated accuracy with unchanged edit-following, contamination improves recall of reasoning steps rather than bypassing them. That would contradict the common assumption in this literature that memorized answers come with decorative explanations.
References
- How Likely Do LLMs with CoT Mimic Human Reasoning?
- ConStat: Performance-Based Contamination Detection in Large Language Models
- DCR: Quantifying Data Contamination in LLMs Evaluation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Measuring Faithfulness in Chain-of-Thought Reasoning
- FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness
- FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning
- Ask, Don’t Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
- FaiRR: Faithful and Robust Deductive Reasoning over Natural Language
- FaithformBench: Benchmarking Faithfulness of Mathematical Chain-of-Thought Autoformalisation
- Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation