When a language model has seen a benchmark question and its answer during training, it can produce the right answer without doing the reasoning. This note proposes catching that by making a small, programmatically verified edit to an intermediate quantity in the problem, an edit that must change the correct answer, and checking whether the model still returns the old answer. The idea borrows a lesson from graph geometry. In a graph, a single spurious "shortcut" edge between two distant regions collapses shortest-path distances, so the computed route ignores the real structure in between. Memorizing a question-to-answer mapping is the same kind of shortcut through a multi-step reasoning chain. Its signature is that the answer stops responding to changes in the intermediate steps it skipped. We call this answer inertia. We measure it on reasoning graphs extracted from benchmark items, and compare it against freshly generated problems with the same graph shape, so the test needs no clean reference model and no token probabilities.
The timing matters. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that when a model is contaminated during supervised fine-tuning and then trained further with clean reinforcement learning, standard memorization detectors drop toward chance. Loss, Min-K% and LiRA all fall to about 50–58% AUROC. The authors explicitly call for detection methods that go "beyond simple log-probabilities." Answer inertia is a behavioral test of exactly that kind, and it runs on text-only APIs.
Core Claim
Treat a benchmark item as a directed acyclic graph (DAG), in which nodes are quantities or premises and edges are the operations that combine them, following DARG and DyVal.
For each item, we create counterfactual versions by changing the value of one leaf node. If the changed node is an ancestor of the answer, the edit lies on the path and the correct answer changes. If not, it lies off the path and the answer stays the same. For an on-path counterfactual set $C_{on}$ of an item with original answer $a$, define the stuck rate
$$S = \frac{1}{|C_{on}|}\sum_{c \in C_{on}} \mathbb{1}[\hat{y}(c) = a],$$
where $\hat{y}(c)$ is the model's answer to counterfactual $c$. A model that reasons can still make errors on $c$. Those errors are spread over many wrong values, so for arithmetic tasks it lands on exactly $a$ only by coincidence. A model following a memorized shortcut lands on $a$ specifically.
The contamination statistic for a benchmark is
$$\Delta = \bar S_{bench} - \bar S_{ref},$$
where $\bar S_{ref}$ is the mean stuck rate on fresh DyVal-style instances. These instances are matched to the benchmark items in depth, width, and numerical range. We test significance by permutation over items.
The graph lens also makes a second, sharper prediction. If memorization acts as a question-to-answer shortcut, stuck rate should be highest for edits far from the answer node and small in surface form, such as changing one leaf number. It should be lowest for edits adjacent to the answer node. A model that genuinely reasons should show the opposite gradient, or no gradient: its errors grow with the number of steps to propagate, but it does not return to the original answer. This depth profile, rather than the scalar alone, is what separates memorization from ordinary difficulty.
How the Two Directions Relate
The shortest-path literature supplies the diagnostic idea rather than an algorithm we run.
Several papers show that single-path metrics are discontinuous under noise, which is why we do not trust single pass/fail outcomes. Yu et al. and Chehreghani show that sparse bridge outliers merge distinct clusters under path-based distances. On Order-Constrained Transitive Distance Clustering shows that minimax paths overfit to chains of tiny gaps.
The most direct inspiration is Stability and Continuity of Centrality Measures in Weighted Graphs. That paper proves betweenness centrality, which depends on a single shortest path, is discontinuous, and replaces it with a stable measure built from detour costs: how much distances grow when a node is removed. Our per-node counterfactual edits are the reasoning-graph version of that node ablation. Aggregating the response across many nodes is what makes the statistic stable rather than jumpy.
Persistent Homology for High-dimensional Data Based on Spectral Methods supports the same design choice from another angle. It defeats shortcut edges by integrating over all paths through effective resistance, which suggests averaging over many perturbations rather than reading one.
The contamination literature supplies the problem and the constraints. Each existing detector has a specific limitation that our test avoids:
- ConStat detects contamination as an accuracy gap on rephrased benchmarks, measured against clean reference models. It fails when no clean reference exists.
- PaCoST uses paired confidence tests and needs model confidence scores.
- DCR fuses multiple levels of contamination evidence into an accuracy adjustment.
None of the three looks at which wrong answer the model gives under a meaning-changing edit, which is the evidence answer inertia is built on.
DyVal and DARG are the closest work. They grow reasoning graphs in depth and width to make fresh, harder problems, and report accuracy drops; DARG reports that Claude-3-Opus loses over 50% accuracy within four added depth steps. They do not hold complexity fixed and edit individual nodes, and they do not classify errors by whether the model returned the memorized answer. An accuracy drop under added depth mixes contamination with difficulty. A same-shape, one-node edit that pulls the model back to the original answer is far harder to explain as difficulty.
Research Program
Graph extraction. We extract a DAG for each item in GSM8K, BBH Navigate, and BBH Dyck using an open model, so the benchmark instances do not inherit biases from a closed model, a concern DARG raises. Labels are verified with code, as in DARG. We keep an item only if executing its graph reproduces the official answer.
Counterfactual generation. For each kept item, we create up to eight one-leaf counterfactuals per node class (on-path and off-path) and record each edited node's hop distance to the answer node. Edits are rendered back into text by rewriting only the edited number or premise. This keeps surface change minimal and the rest of the wording verbatim, and it is deliberate: a shortcut keyed on surface text should survive such an edit.
Reference set. We generate the reference pool with DyVal's generator, using the graph statistics of each benchmark item, and perturb it the same way.
Outputs. Per model and benchmark we report the scalar $\Delta$, the stuck-rate-by-depth profile, and an instance-level score: the stuck rate of that item's counterfactuals, used for instance-level AUROC.
Baselines
Naive reference: no counterfactuals. The simplest reference is plain accuracy on the original benchmark compared with accuracy on fresh DyVal instances of matched shape. This number conflates contamination with any difference in difficulty between benchmark and generated items, so beating it means $\Delta$ ranks known-contaminated models above clean ones where the raw accuracy gap does not.
ConStat. This is the strongest existing behavioral method, because it is also black-box and performance-based. ConStat reports 98.4% detection in its controlled setting, against at best 71% for memorization baselines. It also reports a gap of $\delta \approx 8.3\%$ for Mistral-7b-v0.1 on GSM8K. We run ConStat on the same items and the same contaminated models. It receives the same budget of model queries we use for counterfactuals, its rephrasings come from the same generator model, and its reference models are drawn from the clean controls described below.
Beating ConStat has two parts. First, matching or exceeding its detection accuracy in the controlled setting. Second, staying significant in the setting where every available reference model is itself contaminated, which ConStat's design cannot handle by construction.
Log-probability detectors. Min-K%, LiRA and loss are run as in the Fragility paper, which reports LiRA at 89.1% AUROC before reinforcement learning, a drop of about 14% after one epoch of GRPO, and 50–58% AUROC after full reasoning-model training.
PaCoST. We run PaCoST wherever confidence scores are available. The file gives no per-benchmark effect size for it; it reports only significance at $p<0.05$.
Ablations. The ablation that isolates the graph ingredient replaces graph-aware edits with unstructured edits: the same number of random number changes in the text, with labels recomputed by the same code verifier. If unstructured edits give the same $\Delta$ and the same depth profile, the reasoning-graph structure adds nothing. A second ablation drops the on/off-path distinction and pools all edits.
Why not just…?
Why not just use ConStat with DARG-perturbed variants? This is the real threat. An accuracy gap between original and perturbed items may already detect most contamination. Our argument for something more is twofold. ConStat measures a drop, which harder or awkwardly worded variants also produce. And it needs clean reference models, which is fragile when a whole model family shares the same contaminated data. Answer inertia counts one specific error, reverting to the old answer, and calibrates against generated problems rather than other models. The head-to-head below settles it. If ConStat matches us in every setting, including the all-contaminated-references one, we will report that the extra machinery is unnecessary.
Why not just check whether a perturbed question still gets the original answer, without any graph? That is the unstructured-edit ablation. Without a graph we cannot guarantee that an edit changes the answer, and we cannot tell on-path from off-path edits. That matters because off-path edits should leave the answer unchanged, so returning the original answer there is correct rather than suspicious. We also cannot compute the depth profile, which carries the shortcut prediction. We expect the unstructured probe to be noisier, but this is an empirical claim we test.
Why not just retire static benchmarks and evaluate only on fresh dynamic ones like DyVal? For new evaluations this is reasonable. It does not, however, audit the published scores that leaderboards and papers already rely on. DyVal itself shows those scores can mislead: phi-1.5 and WizardMath-13B score nearly zero on its tasks. Our test reuses DyVal as a calibration source, not a replacement.
Why not just use better log-probability detectors? The Fragility paper shows that reinforcement learning after contamination specifically shrinks the probability gap these detectors depend on. Our hypothesis is that the behavioral shortcut survives even when the probability signal does not. If it does not survive, that is itself a finding: the contamination would have become behaviorally benign on this probe.
Proposed Experiments
Controlled contamination. Following the Fragility paper's setup, we fine-tune Qwen2.5-7B, Llama-3-8B and DeepSeek-R1-Distill-Llama-8B on a random half of GSM8K test items. There are three conditions: answer only, question with full chain-of-thought, and paraphrased question with chain-of-thought. Each contaminated model is then trained either with or without a subsequent clean GRPO stage. The held-out half provides instance-level negatives, and uncontaminated copies trained on matched clean data provide model-level controls. The same procedure is applied to BBH Navigate and Dyck.
Metrics. We report instance-level and model-level AUROC, $\Delta$ with permutation $p$-values, and accuracy inflation on originals versus counterfactuals.
Decisive check 1: survival after reinforcement learning. The first test is whether instance-level AUROC from stuck rate stays clearly above the 50–58% band after GRPO, while log-probability detectors fall into that band on the same models.
Decisive check 2: the depth gradient. The second test is whether contaminated models show the predicted gradient, with stuck rate rising as an edit moves farther from the answer, and clean models do not. If contaminated models show elevated stuck rate but no gradient, the shortcut interpretation is wrong, even if the detector works.
Decisive check 3: the chain-of-thought falsification case. The third test covers chain-of-thought contamination, where the Fragility paper says models learn generalizable reasoning templates. If stuck rate shows no excess there, but accuracy inflation (11.7% in their reasoning-model setting) also disappears on counterfactuals, then that contamination does not transfer to our probe. We would report this plainly as a boundary of the method.
Natural audit. Finally, we compute $\Delta$ for public open models on GSM8K and ask whether models ConStat flags, such as Mistral-7b-v0.1, also show positive $\Delta$ where clean-reference methods have nothing to compare against.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Transitive Distance Clustering with K-Means Duality
- Classification with Minimax Distance Measures
- On Order-Constrained Transitive Distance Clustering
- Stability and continuity of centrality measures in weighted graphs
- Persistent Homology for High-dimensional Data Based on Spectral Methods
- ConStat: Performance-Based Contamination Detection in Large Language Models
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- DCR: Quantifying Data Contamination in LLMs Evaluation