Fractured Chain-of-Thought Reasoning

Baohao Liao, Hanze Dong, Yuhui Xu, Doyen Sahoo, Christof Monz, Junnan Li, Caiming XiongSubmission to ICLR 2026 · Rejected

TL;DR

Long Chain-of-Thought (CoT) reasoning models deliver impressive accuracy but waste massive token budgets on excessive "overthinking." This paper introduces Fractured Sampling, an inference strategy that interrupts reasoning at intermediate depths to generate final answers early, boosting Pass@k efficiency and cutting token consumption by ~20% via early stopping without sacrificing accuracy.

What Problem Does This Paper Address?

Reasoning-focused large language models (such as DeepSeek-R1 and OpenAI's o1) achieve state-of-the-art problem solving by generating extended reasoning traces—often thousands of tokens of self-reflection, planning, and error correction. However, combining these long traces with test-time scaling techniques (like self-consistency or Best-of-$N$ sampling) pushes computational costs and latency to impractical levels.

The authors investigate a fundamental question: Are complete, exhaustive CoT traces truly necessary to arrive at the correct answer? They observe that models frequently pinpoint the crucial insights early, meaning subsequent reasoning tokens contribute little—and can even lead to error-inducing overthinking.

Methods

The authors propose Fractured Sampling, a framework that unifies test-time sampling across three orthogonal dimensions:

  1. Trajectory Diversity ($n$): The number of independent, full-length reasoning paths sampled (standard parallel CoT).
  2. Reasoning Depth Diversity ($H$): Segmenting a thinking trace into $H$ partial prefixes (intermediate depths) and directly generating answers from each stage.
  3. Solution Diversity ($m$): Generating $m$ distinct final solutions given a specific reasoning prefix.
       [Prompt x]
           │
     ┌─────┴─────┐ ... (n independent trajectories)
     │ Trajectory│
     ├── Step 1 ─────────► [Solution 1 ... m]
     ├── Step 2 ─────────► [Solution 1 ... m]
     └── Step H ─────────► [Solution 1 ... m]

Using this setup, the authors introduce two inference enhancements:

  • Denoised Selection: When aggregating with a Process Reward Model (PRM) or Majority Voting, early reasoning steps ($H=1$ to $11$) are noisy; discarding them and keeping only the final depths (e.g., the last four segments, denoted $H=-4$) sharply improves accuracy.
  • Consistency-Based Early Stopping: During generation, the model pauses at set intervals (e.g., token 6144, then every 2048 tokens) to generate a quick final answer candidate. If the same answer appears consistently across checkpoints, reasoning terminates early.

Datasets

Experiments evaluate math and scientific reasoning across five challenging benchmarks:

  • MATH500 Level 5 (Hardest subset of MATH)
  • AIME 2024 & AIME 2025 (American Invitational Mathematics Examination)
  • AIMO2 (AI Mathematical Olympiad progress prize 2 reference set)
  • GPQA Diamond (Graduate-level Google-proof science questions)

Evaluations span multiple open reasoning models: the DeepSeek-R1 family (1.5B, 7B, 14B, and 671B), DeepScaleR-1.5B, Skywork-OR1-7B, Qwen3-1.7B, and GPT-OSS-20B. No new datasets were introduced.

Key Insights and Ideas

  • Thinking Dominates Compute: Token breakdown analyses show that intermediate reasoning accounts for the vast majority of the token budget; the final answer text is minuscule by comparison.
  • Decorrelated Error Modes Across Depths: Failures at different reasoning depths exhibit zero or negative pairwise covariance (especially on GPQA). Truncating at multiple depths decorrelates errors, making it far less likely that all sampled branches fail simultaneously compared to sampling multiple answers from the same full trace.
  • Log-Linear Scaling Favors Depth: Plotting Pass@k against token budget shows log-linear scaling across all three axes, but the depth dimension ($H$) yields the steepest slope ($C_H \ge \max\{C_n, C_m\}$). Branching across intermediate steps provides a higher return-per-token than merely sampling more full trajectories.

Key Results

  • Pass@k Efficiency: Fractured sampling across intermediate depths consistently shifts the compute–performance Pareto frontier upward, achieving higher Pass@k at substantially lower token counts compared to standard vanilla trajectory sampling.
  • Surpassing Larger Models via Selection: On DS-R1-Qwen-7B, combining denoised depth sampling with solution replicates ($H=-4, m=4$) improved Best-of-$N$ (BoN) accuracy from 60.4% to 70.8% and Majority Voting from 66.7% to 71.3%. This 7B configuration outperformed standard sampling on the 14B model (68.3% BoN, 68.0% Maj).
  • Early Stopping Savings: Checking prediction agreement every 2048 tokens reduced token generation by ~20% across benchmarks while matching or improving accuracy (e.g., a +2.9% average accuracy increase on DeepScaleR-1.5B).

Limitations and Future Work

  • Requires White-Box Prefix Access: Fractured sampling requires fine-grained control over intermediate token state injection, making it difficult to use with black-box commercial APIs.
  • Model Architecture Dependency: The strategy relies heavily on models trained to produce structured, explicit CoT traces; models lacking extended natural reasoning chains may see diminished returns.
  • Future Directions: The authors highlight integrating fractured sampling into Reinforcement Learning (RL) rollout pipelines, where high Pass@k under tight token budgets is crucial for training efficiency.

Summary updated

Abstract

Inference-time scaling techniques have significantly bolstered the reasoning capabilities of large language models (LLMs) by harnessing additional computational effort at inference without retraining. Similarly, Chain-of-Thought (CoT) prompting and its extension, Long CoT, improve accuracy by generating rich intermediate reasoning trajectories, but these approaches incur substantial token costs that impede their deployment in latency-sensitive settings. In this work, we first show that truncated CoT, which stops reasoning before completion and directly generates the final answer, often matches full CoT sampling while using dramatically fewer tokens. Building on this insight, we introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling along three orthogonal axes: (1) the number of reasoning trajectories, (2) the number of final solutions per trajectory, and (3) the depth at which reasoning traces are truncated. Through extensive experiments on five diverse reasoning benchmarks and several model scales, we demonstrate that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget. Our analysis reveals how to allocate computation across these dimensions to maximize performance, paving the way for more efficient and scalable LLM reasoning.
BibTeX
@inproceedings{liao2026fractured,
  author = {Baohao Liao and Hanze Dong and Yuhui Xu and Doyen Sahoo and Christof Monz and Junnan Li and Caiming Xiong},
  title = {{Fractured Chain-of-Thought Reasoning}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=HALgBdKfMS},
}
Figure