TL;DR
Long Chain-of-Thought (CoT) reasoning models deliver impressive accuracy but waste massive token budgets on excessive "overthinking." This paper introduces Fractured Sampling, an inference strategy that interrupts reasoning at intermediate depths to generate final answers early, boosting Pass@k efficiency and cutting token consumption by ~20% via early stopping without sacrificing accuracy.
What Problem Does This Paper Address?
Reasoning-focused large language models (such as DeepSeek-R1 and OpenAI's o1) achieve state-of-the-art problem solving by generating extended reasoning traces—often thousands of tokens of self-reflection, planning, and error correction. However, combining these long traces with test-time scaling techniques (like self-consistency or Best-of-$N$ sampling) pushes computational costs and latency to impractical levels.
The authors investigate a fundamental question: Are complete, exhaustive CoT traces truly necessary to arrive at the correct answer? They observe that models frequently pinpoint the crucial insights early, meaning subsequent reasoning tokens contribute little—and can even lead to error-inducing overthinking.
Methods
The authors propose Fractured Sampling, a framework that unifies test-time sampling across three orthogonal dimensions:
- Trajectory Diversity ($n$): The number of independent, full-length reasoning paths sampled (standard parallel CoT).
- Reasoning Depth Diversity ($H$): Segmenting a thinking trace into $H$ partial prefixes (intermediate depths) and directly generating answers from each stage.
- Solution Diversity ($m$): Generating $m$ distinct final solutions given a specific reasoning prefix.
[Prompt x]
│
┌─────┴─────┐ ... (n independent trajectories)
│ Trajectory│
├── Step 1 ─────────► [Solution 1 ... m]
├── Step 2 ─────────► [Solution 1 ... m]
└── Step H ─────────► [Solution 1 ... m]
Using this setup, the authors introduce two inference enhancements:
- Denoised Selection: When aggregating with a Process Reward Model (PRM) or Majority Voting, early reasoning steps ($H=1$ to $11$) are noisy; discarding them and keeping only the final depths (e.g., the last four segments, denoted $H=-4$) sharply improves accuracy.
- Consistency-Based Early Stopping: During generation, the model pauses at set intervals (e.g., token 6144, then every 2048 tokens) to generate a quick final answer candidate. If the same answer appears consistently across checkpoints, reasoning terminates early.
Datasets
Experiments evaluate math and scientific reasoning across five challenging benchmarks:
- MATH500 Level 5 (Hardest subset of MATH)
- AIME 2024 & AIME 2025 (American Invitational Mathematics Examination)
- AIMO2 (AI Mathematical Olympiad progress prize 2 reference set)
- GPQA Diamond (Graduate-level Google-proof science questions)
Evaluations span multiple open reasoning models: the DeepSeek-R1 family (1.5B, 7B, 14B, and 671B), DeepScaleR-1.5B, Skywork-OR1-7B, Qwen3-1.7B, and GPT-OSS-20B. No new datasets were introduced.
Key Insights and Ideas
- Thinking Dominates Compute: Token breakdown analyses show that intermediate reasoning accounts for the vast majority of the token budget; the final answer text is minuscule by comparison.
- Decorrelated Error Modes Across Depths: Failures at different reasoning depths exhibit zero or negative pairwise covariance (especially on GPQA). Truncating at multiple depths decorrelates errors, making it far less likely that all sampled branches fail simultaneously compared to sampling multiple answers from the same full trace.
- Log-Linear Scaling Favors Depth: Plotting Pass@k against token budget shows log-linear scaling across all three axes, but the depth dimension ($H$) yields the steepest slope ($C_H \ge \max\{C_n, C_m\}$). Branching across intermediate steps provides a higher return-per-token than merely sampling more full trajectories.
Key Results
- Pass@k Efficiency: Fractured sampling across intermediate depths consistently shifts the compute–performance Pareto frontier upward, achieving higher Pass@k at substantially lower token counts compared to standard vanilla trajectory sampling.
- Surpassing Larger Models via Selection: On DS-R1-Qwen-7B, combining denoised depth sampling with solution replicates ($H=-4, m=4$) improved Best-of-$N$ (BoN) accuracy from 60.4% to 70.8% and Majority Voting from 66.7% to 71.3%. This 7B configuration outperformed standard sampling on the 14B model (68.3% BoN, 68.0% Maj).
- Early Stopping Savings: Checking prediction agreement every 2048 tokens reduced token generation by ~20% across benchmarks while matching or improving accuracy (e.g., a +2.9% average accuracy increase on DeepScaleR-1.5B).
Limitations and Future Work
- Requires White-Box Prefix Access: Fractured sampling requires fine-grained control over intermediate token state injection, making it difficult to use with black-box commercial APIs.
- Model Architecture Dependency: The strategy relies heavily on models trained to produce structured, explicit CoT traces; models lacking extended natural reasoning chains may see diminished returns.
- Future Directions: The authors highlight integrating fractured sampling into Reinforcement Learning (RL) rollout pipelines, where high Pass@k under tight token budgets is crucial for training efficiency.
Summary updated