Lost at the Beginning of Reasoning

Baohao Liao, Xinyi Chen, Sara Rajaee, Yuhui Xu, Christian Herold, Anders Søgaard, Maarten de Rijke, Christof MonzSubmission to ICLR 2026 · Withdrawn

TL;DR

While long chain-of-thought (CoT) models appear to self-correct and backtrack, their final predictions are overwhelmingly dictated by the very first step of their reasoning trace. When an initial step goes wrong, models rarely recover—resulting in a ~40% drop in accuracy. Leveraging this vulnerability, the authors introduce an early pruning algorithm that scores candidate first steps with a process reward model (PRM) and discards weak trajectories early, matching standard 64-sample majority voting while slashing inference costs by up to 70%.

What Problem Does This Paper Address?

Modern reasoning models (such as DeepSeek-R1 and Claude 3.7 Sonnet) generate long reasoning traces filled with phrases like "Wait, let me rethink that" or "Alternatively...", leading to a widespread assumption that these systems actively self-correct mistakes.

However, long-chain reasoning comes at massive test-time computational costs and often suffers from "overthinking." This paper investigates where reasoning failures actually originate. Are models genuinely exploring hypotheses and correcting mid-stream errors, or are their reasoning trajectories fundamentally constrained from the start?

Methods

  1. Positional Bias & Step-to-Conclusion Similarity: The authors segmented reasoning traces into distinct logical units (using both GPT-5 and lexical delimiters like "alternatively" or fixed token counts) and measured the cosine embedding similarity (MiniLM and SPLADE) between each intermediate step $t_i$ and the final conclusion $c_T$.

  2. First-Step Ablations and Targeted Perturbations:

  3. They isolated the first step $t_1$, forced the model to conclude immediately ($c_1 = \mathcal{M}(p \text{ <think> } t_1 \text{ </think>})$), and measured how predictive $t_1$'s correctness was for the final answer.
  4. They took verified, correct conclusions, stripped answer markers to serve as the initial step $t'_1$, and injected minimal numerical perturbations (e.g., $a \pm 1$) to test whether models could detect and recover from early mistakes.
  5. Early Pruning via First-Step Filtering (TopM): Instead of generating $N=64$ full reasoning paths for self-consistency (majority voting), the method samples $N=64$ candidate initial steps truncated to a fixed budget (len($t_1^n$) = 512 tokens). A lightweight Process Reward Model (Qwen2.5-Math-PRM-7B) scores each initial step. Only the top $M$ (e.g., $M=16$) candidates are continued to full completion before applying majority voting.

Datasets

The authors evaluated models across five multi-step reasoning benchmarks across math, science, and coding:

  • Mathematics: AIME 2024, AIME 2025, and HMMT (February 2025).
  • Science: GPQA Diamond (graduate-level reasoning QA).
  • Code Generation: LiveCodeBench (problems from Jan–Apr 2025).

No new benchmarks were introduced; all tests used publicly available, competitive evaluation sets.

Key Insights and Ideas

  • The First Step Casts the Die: The semantic similarity between intermediate steps and the final answer is highest at step 1 and drops sharply immediately after. The very first reasoning step sets the trajectory; subsequent steps primarily elaborate on or search locally around this initial framing.
  • Self-Correction Is Largely Overstated: If the first step was incorrect, the final accuracy dropped by an average of 40% across models (43% on challenging problems).
  • Brittleness to Early Errors: In perturbation experiments, injecting a tiny discrepancy ($a \pm 1$) into an otherwise flawless reasoning setup caused massive performance collapses (e.g., DS-R1-Qwen-7B accuracy plunged from 94.8% to 28.5%).
  • Overthinking Even When Correct: Models consumed just as many tokens completing a trace when their first step was already sufficient to answer correctly as they did when working from a flawed initial step.

Key Results

  • Matching Majority Voting with >70% Cost Reduction: Selecting $M=16$ continuations from $N=64$ pruned starts matched or exceeded the performance of standard $N=64$ majority voting across all benchmarks while requiring less than 30% of the token compute ($\sim 0.28\times$ token budget).
  • Outperforming Fixed-Budget Baselines: On AIME/HMMT/GPQA, TopM with $M=16$ consistently outperformed standard unpruned sampling with $N=16$ (e.g., Qwen3-8B gained +3.3% average accuracy over $N=16$, and +1.1% over full $N=64$).
  • Generality Across Models: Trends held across multiple model families and sizes, including DeepSeek-R1-Distill (1.5B to 32B), Qwen3 (1.7B to 235B), Skywork-OR1-7B, Magistral-Small, GPT-OSS-20B, and Claude-3.7-Sonnet.
  • Low Overhead: Scoring a 512-token prefix with a 7B PRM added minimal computational cost, translating directly into a $2.7\times$ to $3.7\times$ end-to-end wall-clock speedup compared to generating 64 full traces.

Limitations and Future Work

  • Scope of Benchmarks: Evaluations focused strictly on structured STEM reasoning (math, code, science). Positional dynamics in open-ended or creative tasks remain unverified.
  • Step Definition Heuristics: Although fixing length to 512 tokens proved effective in practice, arbitrary token budgets do not always correspond to complete logical steps across heterogeneous prompt types.
  • Reliance on External Verifiers: Pruning requires a reliable reward model or verifier to evaluate short initial traces; poor reward model performance on specialized domains could undermine pruning precision.

Summary updated

Abstract

Recent advancements in large language models (LLMs) have significantly advanced complex reasoning capabilities, particularly through extended chain-of-thought (CoT) reasoning that incorporates mechanisms such as backtracking, self-reflection, and self-correction. Despite these developments, the self-correction abilities of LLMs during long CoT reasoning remain underexplored. And recent findings on overthinking suggest that such models often engage in unnecessarily redundant reasoning. In this work, we empirically show that the first reasoning step exerts a disproportionately large influence on the final prediction—errors introduced at this stage can substantially degrade subsequent reasoning quality. This phenomenon is consistently observed across state-of-the-art open- and closed-source reasoning models. Leveraging this insight, we propose an efficient sampling strategy that leverages a reward model to identify and retain high-quality first reasoning steps while discarding suboptimal ones, achieving up to a 70% reduction in inference cost without sacrificing accuracy.
BibTeX
@inproceedings{liao2025lost,
  author = {Baohao Liao and Xinyi Chen and Sara Rajaee and Yuhui Xu and Christian Herold and Anders Søgaard and Maarten de Rijke and Christof Monz},
  title = {{Lost at the Beginning of Reasoning}},
  booktitle = {FoRLM @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=7WeJ9oEm2y},
}
Figure