Memorized Answers Are Cheap: Reasoning-Token Cost Curves as a Contamination Fingerprint

When a reasoning model actually works a problem out, the number of intermediate "thinking" tokens it writes should grow with how hard the problem is. When it has seen the problem during training, it can skip the work. We propose to use this gap as a measurement tool. For each model we fit a curve of reasoning-token cost against intrinsic difficulty on freshly generated problems that the model cannot have seen. Items from a public benchmark that the model answers correctly while spending far fewer tokens than that curve predicts are flagged as likely memorized. Chain-of-thought prompting (asking the model to write out intermediate steps before answering) is what makes this cost visible. The detector needs only output length and a difficulty label, so it works on closed API models that expose neither training data nor token probabilities.

The same gap matters for anyone who serves these models. Cost estimates for reasoning workloads, including the per-token prices assumed by routing systems, are calibrated on public benchmarks. If those benchmarks are partly memorized, the estimates undercount the tokens that fresh user traffic will need. Serving latency is not linear in load: queues saturate, and road-traffic-style congestion curves rise polynomially. A modest undercount of tokens can therefore become a large undercount of latency. We will measure both effects: contamination as a deficit in token cost, and how much convex serving costs amplify that deficit.

Core Claim

Let $d_i$ be a difficulty label for item $i$ that does not depend on the model, such as graph depth in a generated reasoning task or the number of steps in a reference solution. Let $T_i$ be the number of reasoning tokens a model emits on item $i$. On problems generated fresh at evaluation time, we fit a cost curve $\hat T(d)$ to correctly solved items only. We use a flexible monotone fit, such as a power law $a d^k$ or a piecewise form that allows a kink. The contamination score of a benchmark item is its log-residual:

$$ s_i = \log T_i - \log \hat T(d_i). $$

Here $T_i$ is observed on the benchmark item and $\hat T(d_i)$ is the tokens the model spends, on average, when it genuinely solves fresh problems of the same difficulty. Our claim is that memorized items sit at strongly negative $s_i$ and fall off the curve, while genuinely solved items scatter around zero.

The signal adds most over accuracy on items the model gets right in both the original and fresh versions. There an accuracy comparison says nothing, but a memorized shortcut still shows up as a cost deficit.

The second claim is about consequences. Suppose serving latency at load $x$ follows a Bureau of Public Roads-style curve $\ell(x) = t_0\,(1 + \beta (x/C)^p)$. Here $t_0$ is free-flow latency, $C$ is capacity, $\beta$ is a scale constant and $p$ is the exponent. Underestimating token load by a factor $r>1$ then underestimates the congestion term by roughly $r^p$ at high load. This follows directly from the power form. A contamination-induced token deficit therefore does not stay a proportional forecasting error.

Relation to Prior Work

Three papers motivate the proposal by showing that reasoning behaviour changes on unfamiliar problems.

  • DyVal generates reasoning problems from directed acyclic graphs with controlled depth and width. Models strong on static benchmarks, such as phi-1.5 and WizardMath-13B, score nearly zero on these problems. It measures only accuracy. We reuse its generator as our source of uncontaminated, difficulty-labelled problems, and add token cost as the measured quantity.
  • Can Large Language Models Reason? A Characterization via 3-SAT sweeps constraint density and finds GPT-4 writes more tokens at the hard phase transition yet reaches roughly 10% accuracy on the task of finding a satisfying assignment. This shows more tokens do not imply success. That is why we fit the curve on correct items only and read contamination from a deficit, never from an excess.
  • Inverse Scaling in Test-Time Compute shows that letting models reason at their own chosen length can lower accuracy. It asks how accuracy depends on length. We ask how length depends on difficulty and familiarity, and we treat its overthinking tasks as a source of noise the fit must absorb.

Two papers constrain the design.

  • Excessive Reasoning Attack on Reasoning LLMs shows a 10-token suffix can inflate DeepSeek-R1-Distill-Qwen-7B's GSM8K reasoning from 169 to 1,531 tokens without hurting accuracy. Reasoning length is therefore easy to manipulate. Our detector is meant for model evaluators, not for adversarial settings where the model developer is hiding contamination, and we test how a length-padding instruction degrades it.
  • The memorization literature detects contamination through other signals. Studying and improving reasoning in humans and machines uses near-zero token surprise, Snoopy uses pretraining term frequency, and PromptBench uses perturbation. These require token probabilities, corpus access, or population-level accuracy comparisons. Our signal needs only output length. Output length has not been studied as a per-item contamination signal.

Four papers supply the cost-modelling side.

We do not propose a new router. We measure an input these routers all take on trust: the token cost of a query.

Research Program

The first component builds matched item pairs across three settings. Each pair shares a difficulty label but differs in exposure.

  • DyVal tasks (arithmetic, Boolean logic, reachability) span depth tiers D1 to D4, generated through the PromptBench library. Difficulty is the graph depth, which is known exactly.
  • GSM8K items are paired with number-substituted variants. Difficulty is the step count of the reference solution.
  • 3-SAT formulas span constraint density, following the 3-SAT study. Density gives a principled difficulty axis with a known hard region.

The second component creates ground truth by planting contamination. We fine-tune open reasoning models (DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B, both studied in the attack paper) on a random half of a pool of generated items. We vary two factors: the number of exposures (1, 4, 16), and whether training shows answers only or answers with short solutions. Seen and unseen items at every difficulty level are then labelled by construction, so detection quality can be scored exactly.

The third component fits $\hat T(d)$ per model on fresh correct items. It compares a power law, a kinked piecewise fit, and a nonparametric monotone fit. It then scores contamination on seen/unseen held-out items and on public GSM8K. For closed models such as o3-mini, no ground truth exists. There we test whether $s_i$ agrees with the item-level accuracy drop on isomorphs.

The fourth component is a serving simulation. We replay a mixed query stream through a pool of model servers with BPR-style latency at exponents $p \in \{1, 2, 4\}$. A capacity plan and a linear-cost router are calibrated on benchmark token counts, then run on fresh-isomorph token counts. We measure the resulting latency forecasting error and the router's realized cost relative to a planner that knew the true token counts, at loads from light to near capacity.

Baselines

Accuracy gap on fresh isomorphs. This is the naive reference and the strongest existing method. It is the DyVal-style comparison of accuracy on original versus regenerated items, scored per item as correct-on-original-but-wrong-on-variant. It is the strongest competitor because it measures exactly the capability collapse that contamination causes: DyVal reports phi-1.5 and WizardMath-13B scoring "nearly zero" on generated tasks. We run it on the same item pairs, with the same model calls and decoding settings as our method. Our method must beat it in area under the ROC curve (AUROC) on planted contamination overall, and it must also achieve AUROC clearly above 0.5 on the subset of items solved in both versions, where the accuracy gap is undefined by construction.

Token-probability detectors. This baseline scores each item by the model's surprise on the benchmark text, following the near-zero-entropy evidence of Yax et al. It runs only on the open models, with identical items. No reported result found on Lacuna for probability-based detection AUROC on planted contamination in reasoning models. We will report it and the combination of both signals.

Raw length ablation. This isolates the key ingredient: scoring items by $\log T_i$ with no difficulty conditioning. If raw length does as well as the difficulty residual, the cost curve contributes nothing and the claim fails.

Linear-cost planner. For the serving study, the baseline is a linear per-token cost model of the kind BEST-Route assumes. BEST-Route reports a 60% cost reduction within 1% of GPT-4o quality on its own benchmarks; we do not expect to reproduce that figure, only to use its cost assumption. The planner is calibrated on the same traffic trace. It "loses" if its forecast error on fresh traffic grows superlinearly with $p$ while a planner calibrated on isomorph token counts does not.

Why not just…?

Why not just compare accuracy on regenerated versions? Because the accuracy gap is blind whenever the model is right both times. This happens exactly when a capable model has also memorized the item. Such items still inflate apparent efficiency and make cost estimates too low. This alternative is a real threat on the items where the model fails the variant, and the planted-contamination experiment settles it. If the length residual adds no AUROC over accuracy on the both-correct subset, we will report the accuracy gap as sufficient and drop the detection claim, keeping only the cost-amplification result.

Why not just use token probabilities? On open models this may well work as well or better. The combined score is the more likely practical outcome. But the reasoning models whose contamination matters most for serving decisions, such as the o-series, do not expose probabilities on their hidden reasoning. For them, length is often the only per-item signal available. The closed-model agreement test measures how far length alone gets.

Why not just model reasoning cost linearly? At light load, linear models are nearly right, and we expect the amplification effect to vanish there. The Price-of-Stability and Frank-Wolfe results say this stops being true as systems saturate. The load sweep shows at which utilization the linear planner's error diverges. If it never does for realistic $p$, the non-linear framing is unnecessary and we will say so.

Proposed Experiments and Decisive Checks

The first decisive check is planted contamination. The difficulty-conditioned residual $s_i$ should separate seen from unseen items with AUROC substantially above raw length, and above chance on the both-correct subset. Separation should grow with the number of exposures and be stronger when solutions were shown.

The second check is robustness. Detection should survive the overthinking tasks from the inverse-scaling codebase, which add length noise. We will also measure how much a simple instruction to "think at length" erases the deficit, reporting this honestly as the method's evasion cost.

The third check is on natural data. On public GSM8K, the fraction of items flagged by $s_i$ should agree with isomorph accuracy drops on open and closed models. The flagged items should also concentrate where static-benchmark accuracy exceeds fresh-task accuracy most.

The fourth check is the serving simulation. It should show the planner's latency underestimate growing roughly as $r^p$ near capacity, where $r$ is the measured token deficit, and staying near $r$ at light load.

If the first two checks pass, contamination becomes measurable from logs any provider already keeps. If the fourth passes, benchmark-calibrated cost models for reasoning workloads need correcting before they are trusted under load.