Reasoning models now cut their own latency by stopping early. A small probe reads the model's internal state, notices the moment it becomes confident in an answer, and ends the chain of thought there. This works well on average. But a separate body of work shows that confidence is least trustworthy in exactly one situation: when the model, and often its whole family of peers, has settled into the same wrong answer. Our claim is that confidence-triggered early exit saves the most time on precisely those cases, so it turns systematic errors into fast errors. We propose a cheap fix. When the probe wants to stop, a small panel of models from different families checks the provisional answer, all in parallel. If they agree, the model stops. If they disagree, the query is sent to a different model rather than being given more of the same thinking.
The work has two parts. The first is a diagnostic. We test whether the accuracy lost to early exit is concentrated on queries where models share errors, which we call "correlated-hard" queries. The second is a stopping rule built from that diagnosis. The rule should keep most of the latency savings while recovering accuracy on the correlated-hard queries. The open question is how much of the gap a cross-family check can close, given that even different families share errors much of the time.
Core Claim
TERMINATOR trains a one-layer probe on the model's final hidden states. The probe detects a "confidence spike" and inserts the end-of-thinking token, which cuts chain-of-thought length by 14% to 55% and halves wall-clock latency on a GH200 GPU across MATH-500, AIME 2025, HumanEval and GPQA. Its working assumption is that an early, confident answer is a good stand-in for a correct one.
Three papers undercut that assumption on a specific slice of queries:
-
Are More LLM Calls All You Need? defines "hard" queries as those whose infinite-sample majority answer is wrong. On these, repeated sampling only reinforces the error, and voting accuracy on MMLU Physics peaks at 10 to 20 calls before declining. - Complementing Self-Consistency with Cross-Model Disagreement shows that agreement among samples from a single model collapses when that model is confidently wrong. Disagreement across model families recovers the missing uncertainty, giving +0.15 AUROC on HotpotQA.
-
Correlated Errors in Large Language Models finds that wrong HELM model pairs pick the same wrong answer about 60% of the time, against a 33% chance rate, and that this sharing grows as models get more accurate.
These papers motivate the proposal. They imply, though none of them tests it, that a probe reading only the model's own confidence will fire early and wrongly on the correlated-hard slice.
The proposed rule is to exit when the probe fires and a cross-family panel agrees with the provisional answer:
$$\text{exit at } t \iff p_t \ge \tau_p \ \text{and}\ \frac{1}{J}\sum_{j=1}^{J}\mathbb{1}[\,v_j(x, \hat y_t)=\text{agree}\,] \ge \tau_a$$
The symbols are as follows:
- $p_t$ is the probe's confidence at token $t$.
- $\hat y_t$ is the answer extracted from the reasoning so far.
- $v_j$ is the verdict of panel member $j$ out of $J$ members on question $x$.
- $\tau_p$ and $\tau_a$ are the probe and agreement thresholds.
When the panel disagrees, the rule does not resume reasoning. It routes the query to a model from a different family, or abstains if a budget forces it.
The reason for routing rather than continuing comes from the definition of hard queries. On those queries the same model converges to the same wrong answer, so more of its tokens should not help. That is the overthinking trap seen from the other side. The experiments test this choice directly.
How This Differs From Existing Work
The papers closest to this question fall into three groups, and none of them combines cross-model checking with a decision about when to stop reasoning.
The first group handles correlated failure inside agent loops. CAMA collapses correlated memory entries into an effective evidence count and learns Expand, Trace and Stop actions. It works over a shared memory pool, relies on provenance metadata, and is not evaluated under wall-clock limits. ExeCRE applies Dawid–Skene estimation over roughly 1,000 executed inputs to decide which reference code to trust. It reduces GPT-5.2's unnecessary revisions of correct code on LiveCodeBench from 113.2 to 14.0, but adds about 26 seconds per problem and fails when all candidates share the same bug. Both papers support the premise that correlated verifiers waste refinement rounds. Both are too slow to act as a per-token stopping rule.
The second group aggregates evaluator panels. CARE separates true quality from confounders shared across judges, reducing MAE by up to 26.8% over majority voting on UltraFeedback. It assumes quality is the dominant shared factor. We borrow its warning that panels can share confounders, but we do not run a matrix or tensor decomposition at each exit.
The third group improves refinement efficiency without modeling error correlation. Critique-RL trains a critic with frozen discriminability and reports up to 5x compute efficiency over voting, but it keeps the actor fixed. OSCA optimizes how samples are spread across configurations and implicitly treats sample success as independent. Our proposal is the missing piece: a stopping decision that asks whether this particular moment of confidence is likely to be a shared mistake.
Research Program
Labeling queries by difficulty. We first label every evaluation query using a method in the spirit of Chen et al.:
- Draw 32 full-length chain-of-thought samples from the main open-weight reasoning model, and 8 samples each from three or four models of different families.
- Label a query "easy" if the main model's full-length majority answer is correct.
- Label it "self-hard" if the main model's majority is wrong but the cross-family majority is correct.
- Label it "correlated-hard" if both majorities agree on the same wrong answer.
The main model must be open-weight because the probe needs hidden states. The other families are 7–9B models, following Hamidieh et al.
The diagnostic. We train a TERMINATOR-style probe on the main model, using its released recipe where available. We then break down the accuracy change from full-length reasoning to early exit by difficulty class. We also record where the exits land: whether, on hard queries, the probe fires on the first answer hypothesis, or on later false starts, which TERMINATOR reports as its weak point on GPQA.
The gate. The panel members are cheap models from different families. Each receives the question and the provisional answer, and returns one short agree/disagree verdict, capped at a small number of output tokens. All calls run in parallel, so the added latency is about one short forward pass plus one round of network calls. On disagreement, the query goes to the strongest available model from a different family than the main one. We compare this against three other actions: resuming the original reasoning, resampling the main model, and abstaining.
Characterizing failure. The correlated-hard slice where the panel shares the wrong answer is beyond the reach of any agreement-based gate. Kim et al. imply this slice will be large. We report its size explicitly, so the method's ceiling is stated rather than hidden.
Baselines
Full-length reasoning. The naive reference is to let the main model reason to its natural end, with no early exit. It sets the accuracy target and the latency ceiling. All methods run on the same queries, hardware and decoding settings. Beating it means matching its accuracy on every difficulty class while using substantially fewer tokens and less time.
TERMINATOR-style early exit. This is the strongest existing method, as described in TERMINATOR. The file reports a 14% to 55% reduction in chain-of-thought length and halved average latency on a GH200. No per-benchmark accuracy figure for TERMINATOR was found on Lacuna. We will therefore retrain and rerun it ourselves on the same model and data, and tune its threshold over the same grid we use for $\tau_p$.
The gate must do two things to count as beating this baseline:
- Recover a statistically significant share of the accuracy that TERMINATOR loses on the correlated-hard and self-hard slices.
- Keep at least 80% of TERMINATOR's token savings, while keeping 95th-percentile latency within a pre-registered margin of TERMINATOR's.
Stricter probe threshold. This baseline simply raises $\tau_p$, which trades latency for accuracy uniformly across all queries. We sweep it to trace a full accuracy-versus-latency curve, and the gate must sit above that curve.
Same-family panel. This ablation isolates the key ingredient. It uses the same gate, but with the same number of parallel calls drawn from the main model or its own family. It matches the gate's added tokens and latency exactly. If it performs as well as the cross-family panel, family diversity is not doing the work and the central claim fails.
Self-consistency at exit. This baseline draws $J$ parallel short continuations from the main model at the moment the probe fires, and stops only if they agree. It uses the same compute as the gate.
Why not just…?
Why not just sample more and vote? Chen et al. show that voting accuracy rises and then falls with more calls on hard queries, because extra samples concentrate on the shared error. Voting also removes the latency benefit that motivates early exit. We include the self-consistency-at-exit baseline at matched compute. Hamidieh et al.'s finding that 10 same-model samples fail to flag confident errors predicts it will fail on the correlated-hard slice. The breakdown by difficulty class will show whether it does.
Why not just raise the probe threshold? A higher threshold delays every exit, including the many easy exits that are correct. The gate intervenes only where the panel disagrees. This alternative is a real threat if hard-query errors produce visibly weaker confidence spikes. The threshold sweep settles it: if some threshold matches the gate's accuracy at equal latency, the gate adds nothing and we will say so.
Why not just use one strong judge instead of a panel? A single strong judge is cheaper to orchestrate. However, Kim et al. report similarity bias when judges evaluate answers from their own lineage, and CARE shows that judges can share confounders and were fooled over 60% of the time on its adversarial set. We run a single-strong-judge variant at matched latency. If it matches the panel, we adopt it as the simpler method.
Why not just keep reasoning when the check fails? Resuming is the cheapest change to the pipeline. On queries whose full-length majority answer is wrong, though, more of the same model's tokens should not help by definition. We test this by comparing resume, resample and route on the slice where the panel disagrees.
Proposed Experiments
We evaluate on the four benchmarks TERMINATOR uses: MATH-500, AIME 2025, HumanEval and GPQA. We add MMLU Physics, where Chen et al. observed non-monotonic voting, as a slice rich in hard queries. We report the following metrics:
- accuracy overall and per difficulty class;
- generated tokens;
- median and 95th-percentile wall-clock latency, with panel calls timed under real parallel dispatch;
- risk–coverage area for the abstention variant.
The experiment has two decisive checks. The first is the premise: does early exit lose accuracy disproportionately on self-hard and correlated-hard queries compared with easy ones? If its losses are spread evenly, the motivation collapses, and we will report that negative result. The second is the mechanism: at matched added latency, does the cross-family panel recover more hard-slice accuracy than the same-family panel and the self-consistency baseline? Finally, the share of correlated-hard queries where the panel also agrees on the wrong answer measures how far agreement-based stopping can go at all. That number is useful whether it turns out small or large.
References
- TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning
- Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems
- Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
- Correlated Errors in Large Language Models
- Beyond Memory Majority: Latent-Source Reasoning for Multi-Agent Memory Arbitration
- ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation
- CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
- Critique-RL: Training Language Models For Critiquing Through Two-Stage Reinforcement Learning
- Scaling LLM Inference with Optimized Sample Compute Allocation