When Does the Model Decide? Commitment Curves for Detecting Memorized Benchmarks

A model that has seen a benchmark problem during training does not need to reason about it. It already "knows where it is going" before it has finished reading. A model meeting the problem for the first time has to work: its eventual answer stays uncertain for a while and becomes fixed only after some intermediate step. This proposal turns that intuition into a measurement. For each problem, we pause the model at many points, both while it reads the problem statement and while it writes its reasoning, and resample the rest of the output many times. The fraction of continuations that reach the model's modal final answer tells us how committed the model already is at that point. Plotting this fraction against position gives a commitment curve. Our claim is that memorized problems produce curves that saturate early, often before the problem has supplied enough information to determine the answer. Genuinely new problems produce curves with a plateau near one half and then a later rise, like a marble resting on a ridge before it rolls into one valley.

This matters because the current tools for catching contamination are failing just as they are most needed. Likelihood-based detectors that flag training-set membership through unusually low loss fall to chance after reinforcement-learning post-training. Time-based benchmarks can show that accuracy drops after a model's training cutoff, but they cannot say which individual problems were memorized. Commitment curves are a behavioral quantity with a direct meaning from dynamical systems. We expect them to survive post-training better than token probabilities do, because reinforcement learning changes how confident the output distribution is, not the order in which information gets used.

Core Claim

Potential landscape analysis in developmental biology describes cell fate as motion on a surface. Stable cell types are valleys, called attractors. Decision checkpoints are saddle points, where directed movement stops and random fluctuation takes over. Pseudodynamics (Fischer et al.) fits a drift-plus-diffusion model to time-course single-cell data and finds exactly such a saddle at the β-selection checkpoint of T-cell development, where drift falls to near zero. Zhou and Huang point out the field's basic handicap: measuring a cell destroys it, so dynamics have to be inferred from population snapshots. Language models do not have this handicap. We can return to any intermediate state of a generation and rerun it as often as we like. So the central object of transition-path theory, the committor, can be estimated directly instead of inferred. The committor is the probability that a system starting from a given state reaches one basin before another.

Formally, let $x_k$ be the model's state after $k$ tokens, counting problem tokens and then reasoning tokens. Let $a^\star$ be the model's modal final answer on that problem. The committor is

$$q(x_k) = \Pr\left[\text{final answer} = a^\star \mid x_k\right],$$

and we estimate it by drawing $M$ continuations from $x_k$ at a fixed temperature. The claim has three testable parts. First, memorized problems reach $q > 0.9$ at a smaller fraction of the problem-plus-reasoning length than matched novel problems. Second, novel problems show a saddle region, a stretch where $q$ sits near 0.5 and has low drift along a learned one-dimensional coordinate, and memorized problems lack it. Third, both signatures persist after reinforcement-learning post-training, where likelihood detectors collapse.

Relation to Prior Work

Three papers motivate the proposal by documenting the gap. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that one epoch of GRPO, a common reinforcement-learning method for reasoning models, pushes loss and Min-K% detectors toward chance. It also shows that contaminating an already-strong reasoning model with worked solutions leaves all ten detectors it tests near random. LiveCodeBench and LiveBench use problem release dates as a firewall and show accuracy cliffs at training cutoffs. They treat the model as a black box and give aggregate, not per-problem, verdicts. Is Your LLM Outdated? adds a caution: even post-cutoff web text is partly shaped by earlier model outputs, so recency alone is not a clean label.

Two papers supply pieces of the method and are its closest competitors. The prompt-truncation audit in Reasoning or Memorization? found that Qwen2.5-Math-7B answers nearly 40% of MATH-500 from only the first 40% of each problem. That is one point on our commitment curve, the committor at a prompt prefix with the answer scored against ground truth. We extend it into the reasoning phase and read the shape of the curve instead of accuracy at one cut. That extension matters because a model contaminated with worked solutions may have absorbed solution heuristics rather than verbatim problem text. Such a model might fail to complete a truncated prompt while still committing unusually early once reasoning begins. LaRA is the strongest existing detector for the post-training regime. It measures how stiffly hidden representations respond to token blanking and paraphrase, and so asks how the state reacts when the input is perturbed. We ask when the future becomes fixed. These are different questions, and we test whether they carry complementary information.

Two papers constrain the design. CANARY needs a clean copy of the base model to compare against, which evaluators of closed models never have. Our black-box variant needs only sampling with a prefilled prefix. Solve the Loop reports "equilibrium internalization": over training, the network learns to produce its first internal representation already at the fixed point of its refinement loop. This is a warning that early commitment can come from ordinary learning of easy structure, not only from memorization. For that reason every comparison below is difficulty-matched or randomized within a single problem pool.

Research Program

The work has three parts. The first is the estimator. For each problem we choose checkpoints at fixed fractions of the problem statement (20%, 40%, 60%, 80%, 100%) and of the model's own greedy reasoning trace (10% to 90%). From each checkpoint we draw $M=16$ continuations, which gives an empirical committor with a binomial standard error of about 0.12 at its worst. Prompt-prefix checkpoints feed the truncated problem followed by an instruction to solve. Reasoning checkpoints prefill the full problem plus the partial trace. This version needs only sampling access.

The second part is a cheaper surrogate built from the model's internals. For open models we record hidden states at every checkpoint and fit a small regressor that predicts the empirical committor from the middle-layer hidden state. The regressor is trained only on committor targets, never on membership labels, so the auditor stays label-free. The surrogate gives a one-dimensional reaction coordinate $s = \hat q(x)$ that can be evaluated at every token. On ensembles of sampled trajectories projected onto $s$, we estimate a Pseudodynamics-style drift $v(s)$ and diffusion $D(s)$ from the mean and variance of increments:

$$v(s) \approx \mathbb{E}[\Delta s \mid s], \qquad D(s) \approx \tfrac{1}{2}\mathrm{Var}[\Delta s \mid s].$$

The "saddle score" is the length of reasoning spent in the region where $|v|/D$ is small. Following Fischer et al., we predict that this region is long for novel problems and absent for memorized ones.

The third part is the detector. Each problem's features are the position at which $q$ first crosses 0.9, the committor at 60% of the prompt, and the saddle score. These are combined by a fixed rule without supervision, the rank average. A supervised version is used only as an upper reference. We are open to the result that the black-box crossing position alone does all the work. In that case the landscape machinery is an explanation, not a requirement, and we will report it that way.

Baselines

The naive reference is a set of likelihood detectors: sequence loss, Min-K%, and, where a reference model is available, LiRA. These run on the same problems and checkpoints. On The Fragility of Benchmark Contamination Detection in Reasoning Models reports LiRA at 89.1% AUROC after contamination by supervised fine-tuning, falling toward roughly 50% for reference-free methods after one GRPO epoch, and 50–58% average AUROC in the setting where a reasoning model is contaminated with worked solutions. We expect to reproduce this collapse, and the collapse is the reason the proposal exists.

The strongest existing method is LaRA. On 60 samples (members from PRIME-RL, non-members from AIME 2026), LaRA combined with Self-Critique reached ROC-AUC 0.79 and 31–38% true-positive rate at 5% false-positive rate. We will reimplement its three metrics from their published definitions, since the file does not say whether code is available, and run them on the same models, checkpoints and splits as our detector. The compute budget is matched: LaRA's perturbed forward passes and our continuations are counted in generated-plus-processed tokens, and each method is capped at the same total. Beating LaRA means an AUROC above 0.79 and a true-positive rate at 5% false-positive rate above 38% on our post-GRPO split, together with an AUROC drop of less than five points from before to after GRPO, where LaRA's drop is measured in the same run. We also report LaRA plus commitment features, to test complementarity.

The prompt-truncation audit of Reasoning or Memorization? is run at the same 40/60/80% cuts with the same number of samples. It doubles as the key ablation: our detector restricted to prompt-prefix checkpoints. The gap between it and the full detector isolates the contribution of reasoning-phase commitment.

A self-consistency baseline measures agreement of final answers and lexical similarity of reasoning traces across 16 full-length samples from the start. This isolates whether where commitment happens adds anything beyond how much the samples agree.

Finally, a linear probe on hidden states is trained with true membership labels. It is a supervised ceiling and not a competitor, but it tells us how much membership signal the internals contain at all.

Why not just…?

Why not just use loss or Min-K%? They are cheap and would be the right tool if they worked. The Fragility paper shows they do not survive post-training, and our pre- and post-GRPO comparison reproduces that result on our own splits, so this baseline is measured rather than assumed.

Why not just extend the prompt-truncation audit? This is a serious threat, because our detector literally contains it. If truncation alone matches the full detector on the setting where a strong reasoning model is contaminated with worked solutions, the reasoning-phase curve is unnecessary and we will say so. We expect it to fall short there specifically, because the Fragility paper finds that such contamination generalizes to similar unseen problems instead of producing verbatim recall. Verbatim recall is exactly what truncation depends on. The ablation described under Baselines settles the question directly.

Why not just measure how similar repeated samples are? This is the cheapest black-box competitor, and it may work well before post-training. Our concern is that reinforcement learning narrows output diversity across all problems, which would erase a diversity signal for the same reason it erases the likelihood gap. The commitment curve compares positions within one problem's trajectory, so a global narrowing shifts the whole curve but should not remove the difference between early and late commitment. If self-consistency matches us after GRPO, the "where" hypothesis is wrong.

Why not just use LaRA or a hidden-state probe? LaRA is the benchmark to beat, and a supervised probe may beat everything. But a probe needs labels no auditor has, and LaRA needs internal activations. Our black-box variant needs neither, so even parity with LaRA in a sampling-only setting would be useful. Complementarity is tested by combining the two.

Proposed Experiments

The decisive experiment is a controlled contamination study following the Fragility paper's protocol. Two open reasoning-capable models, Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Llama-8B, are each fine-tuned on a random half of a pool of problems. The pool consists of competition problems released after each model's documented cutoff, drawn from LiveBench math and reasoning tasks and from LiveCodeBench code generation, with worked solutions included. The model is then run through one epoch of GRPO on unrelated data. Members and non-members come from the same time window and difficulty pool, so the Solve-the-Loop worry about easy problems is controlled by randomization. The primary metric is AUROC and true-positive rate at 5% false-positive rate before and after GRPO, for every method under the matched token budget.

The second experiment tests temporal validity in the wild. On LiveCodeBench, we compare commitment curves for problems released before and after each open model's cutoff, restricted to problems the model solves and stratified by platform difficulty. The prediction is that the fraction of problems with early commitment drops at the cutoff date, sharpening LiveCodeBench's aggregate accuracy cliff into per-problem flags. Following Wu et al., we also compare Qwen2.5-Math-7B against Llama3.1-8B on MATH-500, where contamination of the former is already established.

The negative control uses DyVal graph-structured problems and RandomCalculation arithmetic, which cannot be memorized as instances. Any early commitment there counts as a false positive and calibrates the detector's threshold. DyVal's difficulty levels D1–D4 also let us test whether novel problems show longer saddle regions as difficulty rises, as the landscape picture predicts.

The claim fails outright if, in the controlled study, commitment position does not separate members from non-members after GRPO better than the Fragility paper's collapsed detectors. It also fails if early commitment on DyVal and RandomCalculation is as common as on contaminated items. The landscape-specific claim fails, while the simpler black-box detector survives, if drift-and-diffusion saddle scores add nothing beyond the 0.9 crossing position.