The Signal is in the Steps: Local Scoring for Reasoning Data Selection

Hoang Anh Just, Myeongseob Ko, Ruoxi Jia

TL;DR

When distilling long-form reasoning from large teacher LLMs into smaller student models, standard data selection methods score entire solutions based on global probability, inadvertently favoring verbose and repetitive answers over logical precision. This paper introduces Local Average Log Probability (LALP), a method that evaluates candidate solutions step-by-step using a narrow context window to measure local justification rather than global fluency. By focusing on step-level learnability, LALP improves student model downstream accuracy by up to 9.4% across math, science, and coding benchmarks.

What Problem Does This Paper Address?

Supervised fine-tuning (SFT) via knowledge distillation from powerful teacher models (like DeepSeek-R1 or QwQ) is a primary strategy for embedding multi-step reasoning into smaller, deployable student models. However, modern reasoning pipelines often have access to multiple teachers, each capable of generating multiple correct but stylistically diverse solutions to the same prompt.

Prior state-of-the-art selection methods, such as GRAPE, select training data by calculating the Global Average Log Probability (GALP) of the entire response under the student model, prioritizing solutions that feel "natural" to the student. While this works well for short answers, the authors discover that GALP completely breaks down for long-form reasoning traces from heterogeneous teachers. It falls into a "fluency trap," favoring verbose, repetitive, or structurally bloated answers simply because they contain highly predictable filler text, rather than selecting the most instructionally valuable reasoning steps.

Methods

To address the granularity mismatch of global scoring, the authors propose Local Average Log Probability (LALP). The core idea is to evaluate whether each individual reasoning step is logically justified by its immediate premises, rather than whether the entire document reads smoothly.

The LALP pipeline operates as follows:

  1. Step Segmentation: A candidate response $y$ is partitioned into a sequence of discrete reasoning steps $(s_1, s_2, \dots, s_p)$ using a capable long-context LLM (GLM-4.5-Air).
  2. Local Window Conditioning: Instead of conditioning a token's probability on the entire preceding history, LALP scores each step $si$ by conditioning only on the prompt $x$ and a small window ($k$) of immediately preceding steps.
  3. Equal Step Weighting: The token-averaged log probabilities of the steps are averaged uniformly: $$\text{LALP}(y \mid x) = \frac{1}{p} \sum_{i=1}^{p} \text{LocalLP}(s_i)$$ This ensures that verbose scaffolding does not dilute or dominate the score of critical reasoning transitions.
Figure 3: Effect of context window on teacher ranking. Mean step-level log probability (LALP) computed with increasing context windows. Small windows (5–25%) yield a teacher ordering that matches downstream performance; large windows (50–75%) converge to the GALP ordering and recover the cross-teacher reversal.

Datasets

The authors validated LALP using the following datasets:

  • Training Datasets:
  • MATH: Filtered to difficulty levels 3–5 (8,890 prompts) for within-teacher sanity checks.
  • LIMO (Less Is More for Reasoning): A curated set of 817 high-quality prompts used for the mixed-teacher distillation experiments.
  • Evaluation Benchmarks:
  • Mathematics: A diverse suite including MATH500, AIME 2025, AMC 2023, MINERVA, KAOYAN, OlympiadBench, and CN_MATH_2024.
  • Science Reasoning: GPQA-Diamond (expert-level biology, physics, and chemistry).
  • Code Generation: LiveCodeBench v2.

Key Insights and Ideas

  • Step-Compositional Generalization: By analyzing student model representations, the authors found that full test trajectories rarely match training trajectories, but individual reasoning steps are densely covered by the training data (showing a 23% coverage gap in cosine similarity). Students generalize by recombining reusable local steps, meaning data selection should target step-level learnability.
Figure 2: Per-test-point coverage distributions for AIME 2025 problems. Each violin shows the distribution of cosine similarities between a test point and training data. Red (Global): Trajectory- level embeddings show sparse coverage with high variance across test points (mean=0.760, range 0.68–0.83). Many test problems have no close training match. Blue (Local): Step-level embeddings exhibit dense, consistent coverage (mean=0.935, p10=0.89) with tight distributions per test point, allowing reasoning step to find a similar training step. This 23% coverage gap confirms that step-level representations generalize reliably while full trajectories do not.
  • The Fluency Trap: Global scoring methods assign massive probability mass to highly predictable discourse tokens (e.g., "Okay", "So", "Therefore") which dominate the average score, diluting the signal from sparse, low-probability, reasoning-critical tokens. LALP shifts attribution mass away from discourse fillers (from 42% down to 19%) and toward math/symbolic tokens.
  • Resisting Self-Reinforcement: In long contexts, LLMs suffer from an effect where repeating an intermediate value or phrase makes the model increasingly confident in it, regardless of correctness. LALP's local context truncation prevents this artificial inflation of global log probabilities.

Key Results

  • Teacher Selection: LALP successfully predicts the best teacher dataset before fine-tuning (ranking QwQ-32B > DeepSeek-R1 > Qwen3-32B), whereas GALP completely inverts this ranking.
Cross-teacher ranking reversal under global likelihood versus LALP.
  • Per-Prompt Curation: When selecting the best response per prompt from a mixed-teacher pool, LALP outperforms GALP by 9.4% in downstream accuracy on a Qwen2.5-32B-Instruct student model. Notably, the LALP-curated mixed dataset even outperformed training exclusively on the single best teacher.
  • Cross-Domain Transfer: The benefits of LALP extend beyond mathematics; models trained on LIMO math data selected via LALP showed a +9.1 point improvement on GPQA-Diamond (Science) and a +2.9 point gain on LiveCodeBench-Hard (Coding) compared to GALP.

Limitations and Future Work

The primary limitation acknowledged by the authors is that LALP evaluates standalone, local step justifiability and cannot verify long-range consistency. If a model makes an early logical error, subsequent steps that correctly follow from that false premise will still receive a high LALP score. The authors suggest that future work could explore hybrid approaches that combine local step scoring with global coherence checks to ensure both local precision and global consistency.

Summary updated

Abstract

Distilling long-form reasoning from teacher models into smaller students requires selecting which candidate solutions to train on. Recent work argues that one should select responses the student model assigns highest probability, i.e., favoring solutions ``natural'' to the student. However, we find that this approach works within a single teacher but fails when scaling to long reasoning traces from multiple diverse teachers. We identify a key cause: this approach scores entire solutions, but students generalize by recombining familiar reasoning steps, not by memorizing complete solutions. Full-trajectory scoring optimizes the wrong target; it rewards global fluency while the transferable signal lies in local step transitions. We propose Local Average Log Probability (LALP), which scores each reasoning step using only a small window of preceding context, measuring whether each step is justified by its immediate premises rather than whether the full response looks natural to the student. LALP enables two practical use cases: selecting the best teacher before fine-tuning and curating training data from diverse teacher pools. Across math, coding, and science reasoning tasks, LALP consistently improves accuracy when selecting the most natural solutions by a large margin.
BibTeX
@inproceedings{just2026signal,
  author = {Hoang Anh Just and Myeongseob Ko and Ruoxi Jia},
  title = {{The Signal is in the Steps: Local Scoring for Reasoning Data Selection}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=GcB3a6IonG},
}
Figure