Thinking Until the Answer Stops Moving: Verifier-Free Control of Reasoning Length

Reasoning language models often keep "thinking" long after they have effectively settled on an answer. Sometimes they even argue themselves out of a correct one. The methods that fix this by training mostly depend on an automatic checker that marks each answer right or wrong. That checker is available for math and code, but not for essays, analysis, or long-form answers. This proposal trains a model to stop based on a signal it can compute about itself. At each pause in its reasoning, we ask how much its would-be final answer just changed. We reward the model for spending tokens only while that answer is still shifting abruptly, and we penalize tokens spent after it has stopped moving. Because the signal comes from the model's own predictions rather than a correctness oracle, it applies wherever the model produces an answer, including open-ended tasks.

The design borrows two findings from neuroscience. In the brain, the pupil dilates when an established expectation is suddenly broken, a "model reset," and not when beliefs drift gradually (Violation of Auditory Regularities is Reflected in Pupil Dynamics). We take this as a template for which internal events count as productive work. Separately, dopamine in one striatal region teaches which action to repeat through a signal that ignores reward entirely (Dopaminergic action prediction errors serve as a value-free teaching signal). We take this as a design principle: "how long to think" should be trained by its own value-free channel, kept apart from the channel that judges answer quality.

Core Claim

Overthinking is mostly the tail of tokens after the last abrupt revision of the model's draft answer. If that holds, a length reward built only from the model's own answer-revision dynamics can replace correctness-conditioned length rewards. It should match them on math and keep working when the correctness checker is swapped for a graded, imperfect quality signal on open-ended generation. What is new is the source of the length signal: self-measured draft-answer resets rather than verified correctness. Related work either needs an oracle to decide which length is "right" or acts only at inference time on local confidence.

How This Differs From What Exists

Two training-time methods motivate the gap and define the strongest competitors. DeepCompress uses reinforcement learning to penalize length on problems the model finds easy and reward exploration on hard ones. ShorterBetter rewards closeness to the shortest correct sampled response. Both make the length reward conditional on binary correctness, and ShorterBetter's authors state that their framework "relies on a binary verifier (right/wrong), which is easy for math and code but difficult for open-ended creative writing or subjective reasoning." Our proposal removes correctness from the length channel entirely.

On the unverifiable side, two methods supply the quality signal we reuse but do not control length. JEPO treats the reasoning as a hidden variable and trains against a reference answer string, but it does not control length. ProxyReward turns open-ended long-text evaluation into roughly fifteen automatically generated true/false checklist questions answered by a reader model, scoring only the final text.

Three inference-time methods are close in spirit but differ in mechanism. ThinkBrake stops when the probability of the end-of-thinking token nears that of the top token. LYNX trains hidden-state probes using correctness labels. TrimR uses an external 7B model to stop when the same intermediate answer recurs, and it locks in errors when a wrong answer repeats. Our reset signal is closest to TrimR's notion of "the answer has converged," with three differences: it is a training reward rather than an inference-time cutoff, it is measured as a continuous likelihood rather than string equality, and it needs no second model. Reasoning Is Not a Race supplies the z-score normalization we borrow for detecting abrupt changes, though that work applies it to process reward model scores in math beam search.

Research Program

We first fix notation. A reasoning trace $c$ for prompt $x$ is split into steps at natural pause points such as "Wait" or a paragraph break, capped at a fixed number of segments. Let $y$ be the model's eventual final answer. For each step $k$, we close the thinking block early and compute

$$s_k = \frac{1}{|y|}\log p_\theta\big(y \mid x, c_{\le k}, \texttt{</think>}\big),$$

the average per-token log-likelihood the policy $p_\theta$ assigns to its own final answer if it were forced to answer after step $k$. Here $|y|$ is the answer length in tokens. The quantity works for a number, a program, or an essay, because it only requires scoring text the model already produced. Let $\Delta_k = s_k - s_{k-1}$. A step is a reset when $|\Delta_k|$ exceeds $z$ standard deviations of the trace's running $\Delta$ statistics, following the z-score idea of Reasoning Is Not a Race. Gradual drift does not count, mirroring the pupil finding that gradual updates do not trigger dilation. With $k^\ast$ the last reset step and $L_{>k^\ast}$ the number of tokens after it, the depth reward is

$$R_{\text{depth}} = -\lambda\, \frac{L_{>k^\ast}}{L},$$

where $L$ is total reasoning length and $\lambda$ a weight. Only the converged tail is penalized, so long but still-changing exploration on hard problems costs nothing.

The second ingredient is a separate quality channel, which carries the dopamine-inspired separation of channels. Quality is a binary verifier on math and the ProxyReward checklist score on open-ended prompts. Training uses group relative policy optimization, which compares several sampled responses to the same prompt. Following the observation that action-repetition learning consolidates behavior after value learning has found good behavior (Greenstreet et al.), the depth term enters a response's advantage only when that response's quality is within $\epsilon$ of the best in its group. This generalizes DeepCompress and ShorterBetter's correctness gate to a graded score. The gate exists because the depth reward on its own has an obvious exploit: commit to an answer immediately so that no reset ever happens.

A diagnostic study comes first and determines whether the training study is worth running. We compute $s_k$ curves on DeepSeek-R1-Distill-Qwen 1.5B and 7B traces over MATH500 and Omni-MATH and ask two questions. First, do tokens after $k^\ast$ almost never change correctness? Second, are late resets disproportionately correct-to-wrong flips? The second would be the "talking out of correct answers" failure that ThinkBrake reports.

The pupillometry literature also supplies an evaluation lens. Castner et al. found that expert dentists' pupil dilation scaled with case difficulty (0.051 mm easy versus 0.246 mm hard), while novices sat at a uniform ceiling (0.314 mm median). We define a gear slope: the regression slope of reasoning tokens on labeled difficulty tier, using Omni-MATH's tiers and LiveCodeBench's Easy/Medium/Hard split. A well-regulated model should show a steep slope with a low intercept. The two failures look different: an overthinking model shows a flat, high profile, and an "early-commit" model gaming the depth reward shows a flat, low profile. The overload literature, where pupil size collapses when working memory is exceeded (Effects of Aging and Dual-Task Demands), adds a constraint: more resets is not monotonically better, which is why the reward targets the post-convergence tail rather than counting resets.

Baselines

The naive reference points are the unmodified R1-distilled models, which show the overthinking problem, and the same models trained with a static penalty proportional to length. DeepCompress reports that static penalties "lobotomize" reasoning on hard problems. We expect to reproduce that and include it to calibrate the gear slope.

The strongest existing method on verifiable math is ShorterBetter. Trained on DeepScaleR-preview at 7B, it reports a +7.1% in-domain accuracy gain while cutting length by 62% (ShorterBetter). DeepCompress is a second strong competitor and reports 37.6% fewer tokens with a 5.2-point accuracy gain on AIME 24 at 3B. We retrain both from the same base checkpoints, on DeepScaleR-preview, with the same rollout group size, token budget, and hyperparameter search budget as our method. On math, beating them means matching their accuracy within noise at equal or lower length. The real target is the open-ended setting, where they cannot be run as designed.

On open-ended generation, the reference is ProxyReward training with no length control, using the released pipeline (ProxyReward code). It reports Qwen2.5-7B at 35.07% on ProxyQA. We also report no reported result found on Lacuna for reasoning-length reduction on ProxyQA or WQA by any method. Beating this baseline means holding its ProxyQA and WQA scores while substantially cutting reasoning tokens.

For inference-time comparators, we run ThinkBrake and TrimR on the same checkpoints. LYNX reports up to 12 points on MATH-500, but its probes need correctness labels, so on open-ended data we can only run it in transferred form.

Two ablations isolate the key ingredient. The first, "graded ShorterBetter," takes the shortest response among those within $\epsilon$ of the group's top ProxyReward score, with no reset signal. The second replaces the likelihood-based resets with TrimR-style string-equality resets. We also run a random-reset control to check that the gain is not just generic tail-trimming.

Why not just…?

Why not just plug the ProxyReward score into ShorterBetter as its "correctness"? This is the most serious threat, and it is our main ablation. It may work well on its own. Our argument against it is specific: it penalizes all length relative to the shortest good sample, including exploration that was still changing the answer. Our reward penalizes only the converged tail. If graded ShorterBetter matches our method on ProxyQA quality, token count, and gear slope, the reset signal adds nothing, and we will report the simpler method as the result.

Why not just use an inference-time stopper like ThinkBrake or TrimR? They need no training. However, ThinkBrake's local confidence fails when the model is confidently wrong, and TrimR freezes repeated wrong answers, per their own analyses. Neither is defined for long-form answers without picking an "intermediate answer" format. The comparison on matched checkpoints, and the correct-to-wrong late-reset analysis, will show whether training adds anything beyond a good cutoff.

Why not just translate the dopamine action signal literally and reinforce the model's own high-confidence choices, with no external reward? That amounts to self-confidence or entropy-minimizing reinforcement. It removes the verifier, but it is exactly the reward-free consolidation that the mouse data place after value learning, not instead of it. Without the quality gate it should drive early commitment. We expect it to collapse, and we test this by training with the gate removed, watching for the gear slope going flat and low.

Why not just ask an LLM judge whether the model has thought enough? ProxyReward found general LLM judging noisy enough to hinder training, and the multi-call steering loop of Cognitive Demand Steering adds several model calls per reasoning step. Our signal costs one batched scoring pass over the step prefixes. We will report its wall-clock overhead.

Proposed Experiments

The work proceeds in four stages.

The first stage is the diagnostic. It is decisive: if post-reset tails are not correctness-neutral on MATH500 and Omni-MATH in at least most cases, we stop.

The second stage is math training on DeepScaleR-preview. We evaluate on AIME24/25, MATH500, Omni-MATH and GPQA-Diamond, reporting accuracy, reasoning tokens, and gear slope against the baselines above.

The third stage is open-ended training with ProxyReward-generated checklists, evaluated on ProxyQA and WQA. We report checklist score and tokens. As a guard against the checklist reader being exploited, we compare against held-out checklists produced by a different question-writing model.

The fourth stage is the transfer test, which is the decisive check of the verifier-free claim. We take a ShorterBetter-trained model and our math-trained model, apply each unchanged to ProxyQA and WQA, and compare both to our open-ended-trained model. The claim holds only if training with the reset reward and no verifier beats the transferred verifier-trained length policy on open-ended token efficiency at equal checklist score.

In every stage, the hacking check is the same: a falling gear slope on hard tiers flags early commitment, even when average accuracy looks fine.