TL;DR
When distilling long reasoning chains (10K+ tokens) into smaller LLMs, picking teacher responses using the student's global sequence likelihood fails across multiple teachers. This paper introduces Local Naturalness, a metric that scores teacher responses sentence-by-sentence within a short local context window ($k=4$). Using Local Naturalness to filter mixed-teacher responses boosts a 32B student model's math accuracy by 9.4% over global likelihood selection, outperforming models trained on any single teacher alone.
What Problem Does This Paper Address?
Supervised fine-tuning (SFT) on long chain-of-thought (CoT) traces generated by frontier models (such as DeepSeek-R1 or QwQ-32B) is the primary method for teaching smaller student LLMs to reason. However, when practitioners have access to multiple teachers or multiple candidate outputs per prompt, selecting the best training data is challenging.
Prior work (e.g., GRAPE) selects responses that maximize the student model’s global log-probability, under the premise that data closest to the student's pretraining distribution is easiest to learn. The authors show this heuristic breaks down in multi-teacher setups with long reasoning chains:
- Student models struggle with long contexts (10K–30K tokens), making whole-sequence log-probabilities noisy and poorly calibrated.
- Global likelihood favors stylistic familiarity over actual reasoning quality, ranking weaker teacher data higher simply because it matches the student's surface patterns.
Methods
To fix this degradation over long context windows, the authors propose Local Naturalness:
$$\log P_{\text{local}}(y \mid x; \theta_S) = \frac{1}{p} \sum_{i=1}^p \log P(s_i \mid s_{\max(1, i-k):i-1}, x; \theta_S)$$
- Sentence-Level Decomposition: A response $y$ is split into $p$ logical steps (individual sentences $s_i$).
- Sliding Local Window: Instead of conditioning on the entire preceding token history, each step $s_i$ is evaluated conditioning only on the prompt $x$ and the immediately preceding $k$ sentences (with $k=4$ identified as optimal).
- Two Core Applications:
- Student-Aware Teacher Selection: Aggregating local naturalness across as few as 200 prompts reliably predicts which teacher model will yield the highest downstream student performance.
- Cross-Teacher Response Selection: For each prompt, the response with the highest local naturalness score across a heterogeneous pool of teachers is selected for SFT.
Datasets
- Training Sets:
- LIMO (817 challenging math prompts eliciting long reasoning chains).
- MATH (8,890 prompts, filtered to difficulty levels 3–5).
- OpenCodeReasoning & LeetCode (5,000 prompts for code generalizability).
- Evaluation Benchmarks:
- Mathematics Suite: MATH-500, AIME 2025, AMC 2023, MINERVA, KAOYAN, OLYMPIADBENCH, and CN_MATH_2024.
- General Science: GPQA-Diamond (graduate-level reasoning across biology, chemistry, and physics).
- Coding: LiveCodeBench v2 (evaluated across easy, medium, and hard tiers).
Key Insights and Ideas
- The Fallacy of Training Loss Convergence: Training on responses selected by global log-probability produced the fastest training loss convergence and lowest training loss. However, it led to significantly worse downstream generalization than local scoring. Fast loss decay merely indicated that the model memorized stylistically familiar text rather than acquiring robust reasoning steps.
- Locality of Reasoning: Reasoning inherently relies on sequential, step-by-step local transitions. Student models evaluate "stepping stones" more accurately when unburdened by thousands of tokens of accumulated context.
- Small Context Windows Break Teacher Bias: As the context window expands beyond 50% of the reasoning trace, local scoring collapses back to flawed global rankings. Tight local contexts ($k=4$ sentences) accurately surface high-quality logic even from teachers with lower overall global scores.
- Synergy Across Teachers: The best training set is not homogeneous. Selecting locally optimal responses draws complementary strengths from different teachers, yielding higher downstream accuracy than training exclusively on the single best teacher.
Key Results
- Math Performance: Fine-tuning
Qwen2.5-32B-Instructon LIMO prompts selected with Local Naturalness reached 72.6% average accuracy across seven math benchmarks, outperforming: - Global log-probability selection (63.2%, a +9.4% absolute improvement).
- Random response selection (65.1%).
- The single best teacher baseline (
QwQ-32Balone, at 71.9%). - Outperforming Massive Open Datasets: The 32B student fine-tuned on just 817 locally selected LIMO samples matched or beat prominent open models trained on far larger corpora, including
OpenThinker2-32B(trained on 1.04M examples) andSky-T1-32B-Preview(trained on 17K examples). - Cross-Domain Generalization:
- GPQA-Diamond: Local Naturalness achieved 70.2% pass@1 versus 61.1% for global selection.
- LiveCodeBench v2: Local Naturalness beat global selection on medium (63.3% vs. 58.8%) and hard (26.1% vs. 23.2%) problems.
- Sample-Efficient Teacher Selection: Evaluating local naturalness over just 200 prompts accurately identified the best teacher without running full fine-tuning runs.
Limitations and Future Work
- Fixed Sentence Heuristics: The method uses a static window size ($k=4$ sentences) and treats punctuation-delimited sentences as reasoning steps, which may not always correspond to semantic logical units.
- Offline Filtering: Local naturalness currently acts as an offline filtering step prior to training rather than dynamically interacting with the student during the SFT loop.
- Future Directions: Developing adaptive and content-aware context window sizes, integrating local scoring into online curriculum learning, and extending evaluation to broader non-STEM reasoning tasks.
Summary updated