Reasoning models that "think out loud" before answering, a technique called chain-of-thought (CoT), usually get more accurate. When their input contains misleading text, the extra thinking can backfire. On a science benchmark, turning thinking on for Qwen3-8B raises accuracy by 10–13% on clean inputs. With two misleading snippets in the prompt, accuracy falls to 77.03%, and the model's outputs get much longer as it builds a chain of logic from the false premise (Thinking in a Crowd). Nobody has measured when inside the chain this damage happens. If the model's provisional answer is correct early and then flips to wrong as it rationalizes, the fix is cheap: stop at the right moment. If the model is wrong from the first tokens, no control over generation length can help, and the problem lies in how the model reads its input.
We propose to settle this with depth-resolved answer probing. At regular points in a reasoning chain, we force the model to emit its current answer, which costs one short forward pass on the cached prefix. We record how that provisional answer evolves under clean, irrelevant, and misleading context. We then test a simple early-commit rule: answer at the first depth where the probed answer stops changing. Like any decoding knob, this rule has settings that could be fragile. We therefore treat its sensitivity to those settings, and to temperature, as a first-class result rather than a footnote. Prior studies vary one robustness knob at a time, so their conclusions may depend on decoding settings left at defaults.
Core Claim
Our claim has three parts. First, under misleading context, a large share of thinking-mode errors are correct-to-wrong flips that occur mid-chain, after the model starts engaging with the planted premise. Second, these flips are much rarer under clean context. Third, stopping at the earliest stable probed answer recovers most of the gap between thinking-mode accuracy under misleading context and the best fixed configuration, without hurting clean accuracy and without per-dataset tuning.
All three parts are falsifiable by the flip-rate measurement alone. If wrong-from-the-start errors dominate, we report that the rationalization trap is an input-reading failure. That result would redirect effort toward approaches like context extraction or abstraction rather than budget control.
Research Program
The basic measurement borrows its truncation trick from Fractured Chain-of-Thought Reasoning. That work forces answers from cut-off reasoning prefixes and finds that errors at different depths are decorrelated, so truncated chains often match full ones at equal token cost. It studies only clean math and uses fixed truncation depths.
We reuse the forced-answer mechanism as a measuring instrument under noise. For an input question $x$, context $c$, and reasoning prefix $r_{1:h}$ of length $h$ tokens, the probe distribution is
$$q_h(a) = p_\theta(a \mid x, c, r_{1:h}, \texttt{</think> Answer:}),$$
where $a$ ranges over answer options (or a short extracted answer for open-ended math), and $\theta$ is the frozen model. We probe every 128 tokens. From the sequence $q_{h_1}, q_{h_2}, \dots$ and the gold label, we compute two flip rates per noise condition: correct-to-wrong and wrong-to-correct. We also record the depth at which each flip happens.
The early-commit rule then uses only label-free information. It stops at the first checkpoint after which the top answer under $q_h$ is unchanged for $s$ consecutive probes, and it emits that answer. The rule has one knob, $s$. We report accuracy across the full range of $s$ and across sampling temperatures, using the area-under-the-accuracy–temperature-curve summary that p-less Sampling introduced for decoding robustness.
The hyperparameter-sensitivity literature motivates the program's second strand. AutoRL Hyperparameter Landscapes shows that the best reinforcement-learning settings shift across phases of training, so a static choice is suboptimal. We ask the analogous question for inference. We map accuracy over temperature, truncation rule, and token budget, separately for each noise condition. We then test whether the configuration tuned on clean data lands in a bad region under misleading context.
This study also speaks to an apparent contradiction in the literature. Bounds of Chain-of-Thought Robustness proves that more reasoning steps initially shrink how much small input perturbations move the output, down to a positive floor. That analysis covers small embedding-level perturbations under a linear-attention simplification. Misleading snippets are large semantic perturbations. Our depth curves show directly whether the answer's sensitivity to $c$ decreases or increases with depth in this regime. In this sense the bounds paper constrains our expectations: even a perfect stopping rule should not reach clean accuracy.
Baselines
The naive reference points are full-length thinking and thinking-off, run on the same model and prompts. On SciAux with two misleading snippets, Thinking in a Crowd reports 77.03% for Qwen3-8B with thinking on, 74.87% with thinking off, and 96.22% as the reference baseline. We reproduce these numbers before running anything else.
A stronger reference is an oracle switch that picks thinking on or off per noise condition using test labels. This is an upper bound for any binary policy. Beating it requires per-instance depth selection, not just a mode toggle.
The strongest compute-control competitor is fixed-depth Fractured sampling (Fractured Chain-of-Thought Reasoning), with its depth and vote widths tuned on a clean validation split. It is the closest existing method because it already exploits truncated prefixes. The file gives no reported Fractured-sampling number on SciAux or GSM-Plus; it reports gains only qualitatively on GPQA Diamond and AIME.
Majority voting over sampled full chains, at the same total token count as our method, is the standard test-time scaling answer. We also include p-less decoding (p-less Sampling) with full chains, which is the strongest hyperparameter-free decoder in the file.
Fairness rules apply to every method. All methods see identical prompts and contexts and share a token budget matched to within 5%. Each receives the same tuning budget: a fixed grid of 20 configurations, selected on a clean validation split. Clean-split selection reflects the realistic case where noise type is unknown at deployment.
Two ablations isolate the key ingredient. "Random stop" commits at a random checkpoint drawn to match our method's average depth, which tests whether stability carries information beyond shortening. "Depth-vote" aggregates all probes in the chain, which tests whether committing early beats averaging across depths.
Beating the strongest baseline means two things. Under misleading context, accuracy must exceed both the full-thinking result (77.03%) and clean-tuned Fractured sampling at matched tokens, by a margin outside seed-to-seed variance over five seeds. Clean-context accuracy must stay within one point of full thinking.
Why not just…?
Why not just turn thinking off whenever retrieved context is present? The reported numbers argue against it: thinking-off under misleading context (74.87%) is no better than thinking-on (77.03%), and it gives up the 10–13% clean-input gain (Thinking in a Crowd). The oracle-switch baseline measures exactly how much any binary policy can recover.
Why not just filter the context first, by extracting relevant sentences before reasoning? Generation-side extraction, as in Ext2Gen, which trains the generator to pull out supporting evidence before answering, targets irrelevant chunks. Misleading snippets are relevant by design, so relevance filtering should keep them. This is still a genuine threat. We run a prompted extract-then-reason variant, and if it matches our method under misleading context, we report that budget control is unnecessary.
Why not just sample more chains and vote? BARSA documents a "deceptive majority trap," in which samples converge on the same wrong answer and extra compute entrenches it. A shared misleading premise should produce exactly that correlated error. Matched-token majority voting tests this directly.
Why not just tune temperature or use a hyperparameter-free decoder? It might work, and that is why the landscape map is part of the program. If one static configuration is near-optimal across all noise conditions, the controller adds nothing, and we will say so.
Proposed Experiments
The primary testbed is SciAux (billhdzhao/SciAux), with its clean, irrelevant, and misleading conditions. The models are Qwen3-8B and Hunyuan-7B-Instruct, matching the original study. We add a distilled DeepSeek-R1 model as a third, long-reasoning model.
For math, we use GSM-Plus (Distract), where AbstRaL shows standard CoT collapsing under distractor sentences. AbstRaL itself trains symbolic abstraction with reinforcement learning and an external solver. We include its reported Qwen2.5-0.5B GSM-Symbolic result (34.0% rising to 44.6%) only as context, because it is a training-time method. The file has no reported numeric result for CoT on GSM-Plus (Distract).
ConsisEval (Can LLMs Always Solve Easy Problems) tests whether early commit improves easy-problem consistency on GSM8K pairs, where 7B models score 50–70%. A transfer check uses List Function and MiniARC from the suite in The Curse of CoT, where CoT underperforms direct answering by a relative 20.42%. There, a working stopping rule should commit almost immediately.
The decisive check comes first and is cheap. It is the flip-rate table by condition and depth on SciAux, computed from one set of chains. Correct-to-wrong flips must be both frequent and concentrated under misleading context, or the method has no room to work. The second check is whether stability-based stopping beats random stopping at matched depth. The third is whether our method's accuracy–temperature curve is flatter than that of tuned Fractured sampling, with flatness summarized by its area. We would abandon the method if it wins only at a single tuned value of $s$.
References
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Fractured Chain-of-Thought Reasoning
- $p\textrm{-less}$ Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding
- AutoRL Hyperparameter Landscapes
- Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- BARSA: An Adaptive Test-Time Scaling Strategy for Mathematical Reasoning under Global Compute Budgets
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning