TL;DR
When distilling long-form reasoning from large teacher LLMs into smaller student models, standard data selection methods score entire solutions based on global probability, inadvertently favoring verbose and repetitive answers over logical precision. This paper introduces Local Average Log Probability (LALP), a method that evaluates candidate solutions step-by-step using a narrow context window to measure local justification rather than global fluency. By focusing on step-level learnability, LALP improves student model downstream accuracy by up to 9.4% across math, science, and coding benchmarks.
What Problem Does This Paper Address?
Supervised fine-tuning (SFT) via knowledge distillation from powerful teacher models (like DeepSeek-R1 or QwQ) is a primary strategy for embedding multi-step reasoning into smaller, deployable student models. However, modern reasoning pipelines often have access to multiple teachers, each capable of generating multiple correct but stylistically diverse solutions to the same prompt.
Prior state-of-the-art selection methods, such as GRAPE, select training data by calculating the Global Average Log Probability (GALP) of the entire response under the student model, prioritizing solutions that feel "natural" to the student. While this works well for short answers, the authors discover that GALP completely breaks down for long-form reasoning traces from heterogeneous teachers. It falls into a "fluency trap," favoring verbose, repetitive, or structurally bloated answers simply because they contain highly predictable filler text, rather than selecting the most instructionally valuable reasoning steps.
Methods
To address the granularity mismatch of global scoring, the authors propose Local Average Log Probability (LALP). The core idea is to evaluate whether each individual reasoning step is logically justified by its immediate premises, rather than whether the entire document reads smoothly.
The LALP pipeline operates as follows:
- Step Segmentation: A candidate response $y$ is partitioned into a sequence of discrete reasoning steps $(s_1, s_2, \dots, s_p)$ using a capable long-context LLM (GLM-4.5-Air).
- Local Window Conditioning: Instead of conditioning a token's probability on the entire preceding history, LALP scores each step $si$ by conditioning only on the prompt $x$ and a small window ($k$) of immediately preceding steps.
- Equal Step Weighting: The token-averaged log probabilities of the steps are averaged uniformly: $$\text{LALP}(y \mid x) = \frac{1}{p} \sum_{i=1}^{p} \text{LocalLP}(s_i)$$ This ensures that verbose scaffolding does not dilute or dominate the score of critical reasoning transitions.
Datasets
The authors validated LALP using the following datasets:
- Training Datasets:
- MATH: Filtered to difficulty levels 3–5 (8,890 prompts) for within-teacher sanity checks.
- LIMO (Less Is More for Reasoning): A curated set of 817 high-quality prompts used for the mixed-teacher distillation experiments.
- Evaluation Benchmarks:
- Mathematics: A diverse suite including MATH500, AIME 2025, AMC 2023, MINERVA, KAOYAN, OlympiadBench, and CN_MATH_2024.
- Science Reasoning: GPQA-Diamond (expert-level biology, physics, and chemistry).
- Code Generation: LiveCodeBench v2.
Key Insights and Ideas
- Step-Compositional Generalization: By analyzing student model representations, the authors found that full test trajectories rarely match training trajectories, but individual reasoning steps are densely covered by the training data (showing a 23% coverage gap in cosine similarity). Students generalize by recombining reusable local steps, meaning data selection should target step-level learnability.
- The Fluency Trap: Global scoring methods assign massive probability mass to highly predictable discourse tokens (e.g., "Okay", "So", "Therefore") which dominate the average score, diluting the signal from sparse, low-probability, reasoning-critical tokens. LALP shifts attribution mass away from discourse fillers (from 42% down to 19%) and toward math/symbolic tokens.
- Resisting Self-Reinforcement: In long contexts, LLMs suffer from an effect where repeating an intermediate value or phrase makes the model increasingly confident in it, regardless of correctness. LALP's local context truncation prevents this artificial inflation of global log probabilities.
Key Results
- Teacher Selection: LALP successfully predicts the best teacher dataset before fine-tuning (ranking QwQ-32B > DeepSeek-R1 > Qwen3-32B), whereas GALP completely inverts this ranking.
- Per-Prompt Curation: When selecting the best response per prompt from a mixed-teacher pool, LALP outperforms GALP by 9.4% in downstream accuracy on a Qwen2.5-32B-Instruct student model. Notably, the LALP-curated mixed dataset even outperformed training exclusively on the single best teacher.
- Cross-Domain Transfer: The benefits of LALP extend beyond mathematics; models trained on LIMO math data selected via LALP showed a +9.1 point improvement on GPQA-Diamond (Science) and a +2.9 point gain on LiveCodeBench-Hard (Coding) compared to GALP.
Limitations and Future Work
The primary limitation acknowledged by the authors is that LALP evaluates standalone, local step justifiability and cannot verify long-range consistency. If a model makes an early logical error, subsequent steps that correctly follow from that false premise will still receive a high LALP score. The authors suggest that future work could explore hybrid approaches that combine local step scoring with global coherence checks to ensure both local precision and global consistency.
Summary updated