Large reasoning models often find the right answer early in a long chain of thought, then keep going. The extra tokens cost time and sometimes overwrite a correct answer with a worse one. Current fixes stop generation with hand-set rules: a confidence threshold, a count of hesitation words, or a bandit that picks among a few thresholds. We propose to learn the stopping rule directly by gradient descent on a smoothed version of the quantity we care about, accuracy at a fixed average token budget. The rule is also allowed to return an answer it saw earlier instead of only the latest one. The key observation is simple. Stopping a chain of thought never changes the text written before the stop. So one fully logged reasoning trace already contains the exact outcome of stopping at every possible point. Over such logs, a soft stopping rule gives an expected accuracy and an expected token count that are exactly computable and differentiable, with no reinforcement learning and no off-policy correction.
This connects two lines of work that have not met. Differentiable metric approximation replaces discrete metrics with smooth surrogates, such as sigmoids in place of step functions, so that they can be optimized by gradient. Work on iterative refinement latency has documented how costly multi-pass generation is, but still relies on discrete heuristics to cut that cost. The deliverable is a small, cheap controller that sits on top of a frozen reasoning model. It learns many stopping parameters jointly, includes a "recall" step that picks the best earlier answer, and is judged on hard, unsmoothed accuracy against tokens spent.
Core Claim
A stopping-and-recall controller trained by gradient on exactly computed, smoothed expected accuracy under a token budget will reach a better accuracy–token trade-off than (a) grid-searched single confidence thresholds, (b) bandit-tuned thresholds, and (c) a myopic "is this answer correct?" classifier. The gains should come from two specific ingredients: conditioning on several features jointly, and recall.
What is new is not the use of a sigmoid on an exit threshold. Empowering Adaptive Early-Exit Inference with Latency Awareness already relaxes the exit indicator with a logistic function and optimizes thresholds under an average-latency constraint. It does so for feedforward and LSTM classifiers, with no-recall exits and fixed per-layer thresholds. We carry this into autoregressive reasoning, where stopping points are token positions inside a trajectory whose length varies. We make the stop decision depend on the trajectory's features, and we add recall.
Why Recall, and Why Gradients
Two results set the design. Hold That Exit proves that no "no-recall" policy can guarantee a constant-factor approximation to the optimal accuracy–latency trade-off. A no-recall policy is one where passing an exit is irrevocable and the last prediction is returned. That paper's dynamic-programming RECALL policy fixes this in principle. In practice it loses about 8% accuracy on ImageNet, which the authors attribute to quantizing the loss history into discrete states, and it assumes Markovian transitions between exits.
DEER motivates the reasoning setting. It reports that in about 75% of solutions the key insight appears early. It exits when a branched trial answer's geometric-mean token confidence exceeds a grid-searched threshold, and it discards trial answers that fail the threshold. DEER is therefore exactly a no-recall policy. It also fails on "shy" models whose confidence is low even when they are correct.
Our controller replaces RECALL's discretized state with a continuous learned score. It replaces DEER's single threshold with a feature-conditioned stopping probability. Both are trained end-to-end through a smooth objective.
Method
We run a frozen reasoning model to completion on training questions. At checkpoints $k = 1, \dots, K$ we branch a short trial answer, as DEER does with shared key-value cache, and log three things. The first is a feature vector $x_k$: trial-answer confidence, token entropy, tokens so far, and whether the trial answer agrees with earlier ones. The checkpoints are DEER-style action-transition points or fixed token intervals. The second is $c_k \in \{0,1\}$, the hard correctness of the trial answer. The third is $t_k$, the cumulative token cost including branches.
A small network $g_\theta$ outputs a stop probability $s_k = \sigma(g_\theta(x_{\le k})/\tau)$, where $\sigma$ is the sigmoid and $\tau$ a temperature. The probability of stopping at $k$ is then $\pi_k = s_k \prod_{j<k}(1-s_j)$. A recall head $r_\phi$ scores each earlier trial answer. A softmax over those scores gives weights $w_{jk}$ for returning answer $j$ when stopping at $k$.
Because the prefix up to $k$ is unchanged by stopping, the following per-trajectory objective is exact under the soft policy:
$$ \mathcal{J}(\theta,\phi) = \sum_{k=1}^{K} \pi_k \sum_{j \le k} w_{jk}\, c_j \;-\; \lambda\Big(\sum_{k=1}^{K}\pi_k t_k - B\Big), $$
Here $B$ is the target mean token budget and $\lambda$ a Lagrange multiplier, updated by dual ascent or by the inexact proximal-point penalty method of Li et al. We anneal $\tau$ toward zero and harden both heads at deployment, so that the stop decision and the returned answer are deterministic.
The annealing addresses the fidelity gap that the differentiable-metric literature flags. A sharp sigmoid gives exploding or vanishing gradients, while a flat one optimizes the wrong objective. Learning to Teach with Dynamic Loss Functions and Optimizing Differentiable Relaxations of Coreference Evaluation Metrics are the methodological precedent: smooth probabilistic relaxations of discrete metrics, used instead of reinforcement learning. Le and Titov report peak performance in about 15 hours versus up to 7 days for RL systems.
We avoid the memory problems of unrolled differentiation and the ODE-adjoint machinery of Accelerating Optimization via Differentiable Stopping Time. That paper needed the machinery because changing optimizer parameters changes the whole trajectory. In our setting the trajectory is fixed and only the stopping point moves. We state this contrast explicitly because it is what makes the approach cheap.
The same construction applies to critique–revise loops. Each revision round becomes a checkpoint, and recall becomes a learned, smooth version of the hard "Decide" step in Volcano. That step compares draft and revision, but the loop always runs, raising response time from 2.7 s to 5.8 s.
Research Program
The first phase builds the logging harness and the controller on one open reasoning model, Qwen3-8B, matching the model REFRAIN reports on. We log full trajectories on training splits of MATH and GSM8K, branching trial answers at every checkpoint.
The second phase characterizes the data. It measures how often a correct trial answer appears and is later lost, which is the headroom for recall. It also measures how this varies with difficulty, and how miscalibrated the confidence feature is. DEER's 75% figure suggests the headroom is large, but we will measure it on our own logs.
The third phase trains controllers across a sweep of budgets $B$ and traces hard accuracy against mean tokens.
The fourth phase tests transfer. A controller trained on MATH is evaluated on GPQA-Diamond and HumanEval without retraining. A controller trained on one model is evaluated on a second model with different calibration, to test the shy-model failure.
The fifth phase extends to the multimodal refine loop on POPE and MMHal-Bench, using Volcano's critique–revise rounds as checkpoints.
Baselines
The naive reference points are two. The first runs the model to completion with no early exit, which sets the full-cost accuracy. The second truncates at a fixed token budget, swept over budgets. Statistical Early Stopping argues that fixed truncation fails across domains, and we include it to confirm that on our data.
The strongest existing method on this exact question is DEER. It uses the same trial-answer branching, so we reuse its checkpoints and trial answers, and the comparison isolates the decision rule. DEER reports sequence-length compression of 19.1% to 80.1% with accuracy gains of 0.3% to 5.0% across MATH-500 and related benchmarks. The file does not give a single per-benchmark pair for our model. We will therefore re-run DEER ourselves with its threshold $\lambda$ grid-searched on the same training split, with the same number of evaluations as our controller's tuning budget. We will then compare at matched mean tokens.
REFRAIN is the second strongest competitor because it already adapts thresholds online. It reports a 6.56-point accuracy gain on GPQA-Diamond with Qwen3-8B while using fewer tokens, and 20% to 55% token reduction across benchmarks. We will run it on the same model and benchmarks, including its SBERT overhead in latency.
We also re-implement RECALL from Hold That Exit on the same logged features, discretizing as that paper does. This tests whether smoothing removes the discretization loss.
Beating a baseline means one of two things. Either we reach higher hard accuracy at equal mean tokens across most of the budget range, or we use at least as few tokens at equal accuracy, with differences larger than the seed-to-seed spread over five controller trainings.
Three ablations isolate the ingredients. A no-recall variant always returns the latest trial answer. A single-global-threshold variant learned by gradient should match grid-searched DEER; if it beats it, our gains are an optimization artifact rather than expressiveness. A straight-through hard-threshold variant without temperature annealing tests whether the smoothing schedule matters.
Why not just…?
Why not just grid-search DEER's threshold more carefully? For one scalar threshold, grid search is as good as gradients, and we expect our single-threshold ablation to confirm this. The claim is not that gradients beat grid search on one knob. It is that a rule conditioned on several features, plus recall, beats any single threshold. That rule has too many parameters for grid search or bandits. Li et al. report 7% to 9% gains over single-threshold heuristics in the feedforward case once many thresholds are optimized jointly. If our feature-conditioned controller without recall does not beat a tuned single threshold, the expressiveness argument fails.
Why not just train a classifier that predicts whether the current trial answer is correct, and stop when it is confident? This is the most serious threat. It uses the same features, trains by ordinary cross-entropy, and is easy to deploy. Its weakness is that it is myopic. It ignores both the token cost still to come and whether the answer is likely to improve or degrade. It is also a no-recall policy, which Hold That Exit shows cannot be near-optimal in general. We run it as a baseline with its threshold swept to trace a full curve. If it matches our controller, the honest conclusion is that the differentiable budget objective adds nothing beyond a calibrated correctness probe. We would then report that.
Why not just train the reasoning model with RL and a length penalty? That changes the model and costs a full RL run per budget. Our controller trains in minutes on logged data and leaves the model untouched. The coreference and learning-to-teach results above show relaxations beating RL on cost and stability in other structured settings. That does not prove the same here. We therefore treat length-penalized RL as complementary and test whether our controller still helps on top of an RL-shortened model when one is available.
Why not just use best-of-N with a verifier? Teaching Language Models to Self-Improve by Learning from Language Feedback finds sequential refinement beats best-of-N re-ranking. Best-of-N also spends N times the compute regardless of difficulty, which is the opposite of the goal here.
Proposed Experiments and Decisive Checks
The main experiment plots hard accuracy against mean tokens on MATH-500, GSM8K, GPQA-Diamond, and HumanEval with Qwen3-8B. The controller, DEER, REFRAIN, the correctness classifier, discretized RECALL, and truncation appear on the same axes. We summarize each curve by its area and by accuracy at three matched budgets.
There are four decisive checks. First, does recall recover the trajectories where a correct early answer is later lost? We measure this directly on logs. Second, does the full controller beat the myopic classifier at matched tokens? This is the decisive test of the objective. Third, does smoothing close RECALL's discretization gap on the same features? Fourth, does the controller exit early on a poorly calibrated second model where DEER's threshold fails?
For the multimodal extension, success means keeping Volcano's POPE accuracy of 88.3% and its MMHal-Bench score of 2.64 while cutting mean response time well below its 5.8 s. That would happen by skipping revision on drafts the controller judges already sound.
A clear negative result would also be informative. If the gains collapse to those of a single tuned threshold, then for current reasoning models the stopping problem is effectively one-dimensional.
References
- Empowering Adaptive Early-Exit Inference with Latency Awareness
- Hold That Exit: Near Optimal Early-Exit Inference via Recall
- Dynamic Early Exit in Reasoning Models
- Learning to Teach with Dynamic Loss Functions
- Optimizing Differentiable Relaxations of Coreference Evaluation Metrics
- Accelerating Optimization via Differentiable Stopping Time
- Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision
- Statistical Early Stopping for Reasoning Models
- Teaching Language Models to Self-Improve by Learning from Language Feedback