Distilling Reasoning into Student LLMs: Local Naturalness for Selecting Teacher Data

Hoang Anh Just, Myeongseob Ko, Ruoxi JiaSubmission to ICLR 2026 · Withdrawn

TL;DR

When distilling long reasoning chains (10K+ tokens) into smaller LLMs, picking teacher responses using the student's global sequence likelihood fails across multiple teachers. This paper introduces Local Naturalness, a metric that scores teacher responses sentence-by-sentence within a short local context window ($k=4$). Using Local Naturalness to filter mixed-teacher responses boosts a 32B student model's math accuracy by 9.4% over global likelihood selection, outperforming models trained on any single teacher alone.

What Problem Does This Paper Address?

Supervised fine-tuning (SFT) on long chain-of-thought (CoT) traces generated by frontier models (such as DeepSeek-R1 or QwQ-32B) is the primary method for teaching smaller student LLMs to reason. However, when practitioners have access to multiple teachers or multiple candidate outputs per prompt, selecting the best training data is challenging.

Prior work (e.g., GRAPE) selects responses that maximize the student model’s global log-probability, under the premise that data closest to the student's pretraining distribution is easiest to learn. The authors show this heuristic breaks down in multi-teacher setups with long reasoning chains:

  1. Student models struggle with long contexts (10K–30K tokens), making whole-sequence log-probabilities noisy and poorly calibrated.
  2. Global likelihood favors stylistic familiarity over actual reasoning quality, ranking weaker teacher data higher simply because it matches the student's surface patterns.

Methods

To fix this degradation over long context windows, the authors propose Local Naturalness:

$$\log P_{\text{local}}(y \mid x; \theta_S) = \frac{1}{p} \sum_{i=1}^p \log P(s_i \mid s_{\max(1, i-k):i-1}, x; \theta_S)$$

  • Sentence-Level Decomposition: A response $y$ is split into $p$ logical steps (individual sentences $s_i$).
  • Sliding Local Window: Instead of conditioning on the entire preceding token history, each step $s_i$ is evaluated conditioning only on the prompt $x$ and the immediately preceding $k$ sentences (with $k=4$ identified as optimal).
  • Two Core Applications:
  • Student-Aware Teacher Selection: Aggregating local naturalness across as few as 200 prompts reliably predicts which teacher model will yield the highest downstream student performance.
  • Cross-Teacher Response Selection: For each prompt, the response with the highest local naturalness score across a heterogeneous pool of teachers is selected for SFT.

Datasets

  • Training Sets:
  • LIMO (817 challenging math prompts eliciting long reasoning chains).
  • MATH (8,890 prompts, filtered to difficulty levels 3–5).
  • OpenCodeReasoning & LeetCode (5,000 prompts for code generalizability).
  • Evaluation Benchmarks:
  • Mathematics Suite: MATH-500, AIME 2025, AMC 2023, MINERVA, KAOYAN, OLYMPIADBENCH, and CN_MATH_2024.
  • General Science: GPQA-Diamond (graduate-level reasoning across biology, chemistry, and physics).
  • Coding: LiveCodeBench v2 (evaluated across easy, medium, and hard tiers).

Key Insights and Ideas

  • The Fallacy of Training Loss Convergence: Training on responses selected by global log-probability produced the fastest training loss convergence and lowest training loss. However, it led to significantly worse downstream generalization than local scoring. Fast loss decay merely indicated that the model memorized stylistically familiar text rather than acquiring robust reasoning steps.
  • Locality of Reasoning: Reasoning inherently relies on sequential, step-by-step local transitions. Student models evaluate "stepping stones" more accurately when unburdened by thousands of tokens of accumulated context.
  • Small Context Windows Break Teacher Bias: As the context window expands beyond 50% of the reasoning trace, local scoring collapses back to flawed global rankings. Tight local contexts ($k=4$ sentences) accurately surface high-quality logic even from teachers with lower overall global scores.
  • Synergy Across Teachers: The best training set is not homogeneous. Selecting locally optimal responses draws complementary strengths from different teachers, yielding higher downstream accuracy than training exclusively on the single best teacher.

Key Results

  • Math Performance: Fine-tuning Qwen2.5-32B-Instruct on LIMO prompts selected with Local Naturalness reached 72.6% average accuracy across seven math benchmarks, outperforming:
  • Global log-probability selection (63.2%, a +9.4% absolute improvement).
  • Random response selection (65.1%).
  • The single best teacher baseline (QwQ-32B alone, at 71.9%).
  • Outperforming Massive Open Datasets: The 32B student fine-tuned on just 817 locally selected LIMO samples matched or beat prominent open models trained on far larger corpora, including OpenThinker2-32B (trained on 1.04M examples) and Sky-T1-32B-Preview (trained on 17K examples).
  • Cross-Domain Generalization:
  • GPQA-Diamond: Local Naturalness achieved 70.2% pass@1 versus 61.1% for global selection.
  • LiveCodeBench v2: Local Naturalness beat global selection on medium (63.3% vs. 58.8%) and hard (26.1% vs. 23.2%) problems.
  • Sample-Efficient Teacher Selection: Evaluating local naturalness over just 200 prompts accurately identified the best teacher without running full fine-tuning runs.

Limitations and Future Work

  • Fixed Sentence Heuristics: The method uses a static window size ($k=4$ sentences) and treats punctuation-delimited sentences as reasoning steps, which may not always correspond to semantic logical units.
  • Offline Filtering: Local naturalness currently acts as an offline filtering step prior to training rather than dynamically interacting with the student during the SFT loop.
  • Future Directions: Developing adaptive and content-aware context window sizes, integrating local scoring into online curriculum learning, and extending evaluation to broader non-STEM reasoning tasks.

Summary updated

Abstract

Distilling long reasoning traces (10K+ tokens) from stronger teacher models into smaller student LLMs via supervised fine-tuning (SFT) has emerged as a standard paradigm. This approach is both practical and efficient: it leverages the ease of generating abundant reasoning data from stronger models and provides a direct, data-driven way to teach less capable models better reasoning. While previous work has largely focused on prompt selection with responses from a single teacher, the equally important problem of choosing the best response when multiple teacher outputs are available for a single prompt remains underexplored. This challenge becomes especially important in a multi-teacher setting, where different students may benefit from the outputs of different teachers. This paper fills that gap with a systematic study of response selection for reasoning distillation. We first show that the current method, which picks the response that the student assigns the highest global log-probability (i.e., global "naturalness"), fails when responses come from multiple teachers. In such cases, global naturalness no longer correlates with downstream performance, especially as the reasoning traces from strong teachers become longer. To overcome this limitation, we introduce Local Naturalness, which scores a response by measuring the student’s log-probabilities over short, sequential reasoning steps (e.g., sentences) conditioned only on a small local window. Local Naturalness enables two novel applications: 1) Teacher Selection: Aggregating local scores across prompts reliably identifies the most helpful teacher, whereas global scoring fails completely. 2) Response Selection from a Mixed-Teacher Dataset: When mixing answers from many teachers, Local Naturalness boosts a 32-billion-parameter student’s accuracy on math benchmarks by 9.4% over global- naturalness-based selection, also surpassing the performance achieved by training on data from the single best teacher. These results highlight the power of localized data quality evaluation and data mixing for more effective reasoning distillation.
BibTeX
@inproceedings{just2026distilling,
  author = {Hoang Anh Just and Myeongseob Ko and Ruoxi Jia},
  title = {{Distilling Reasoning into Student LLMs: Local Naturalness for Selecting Teacher Data}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=GcB3a6IonG},
}
Figure