In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners

Jaehoon Kim, Kwangwook Seo, Dongha LeeSubmission to ICLR 2026 · Rejected

TL;DR

Directly fine-tuning small language models on long, complex reasoning traces from massive teachers often backfires, degrading the student's performance. The authors introduce Reverse Speculative Decoding (RSD), where a teacher proposes reasoning steps but the student accepts them only if they fall within its own probability distribution. By filtering out high-surprisal tokens, RSD turns teacher reasoning into student-friendly demonstrations, transforming a 20.5% performance drop into a 4.9% gain on Qwen3-0.6B across standard reasoning benchmarks.

What Problem Does This Paper Address?

Supervised fine-tuning (SFT) on reasoning traces (like DeepSeek-R1 or s1K) has become the standard method for transferring problem-solving capabilities from large teachers to smaller models. While models at the 32B scale benefit dramatically, applying these exact traces to sub-4B models often paradoxically causes severe performance degradation.

The authors attribute this breakdown to distributional misalignment:

  • Reasoning traces generated by massive models contain logical leaps and vocabulary transitions that have near-zero probability under the student’s internal distribution.
  • Forcing a tiny model to recite these "high-surprisal" tokens overwhelms its capacity, creating learning obstacles rather than instructive guidance.

Methods

To smooth the cognitive stride of reasoning traces, the authors propose Reverse Speculative Decoding (RSD) and a hybrid distillation pipeline:

Teacher proposes token y ~ Pt
         │
         ▼
  Is Ps(y) ≥ p_th?
    ├── YES ──► Accept teacher token y
    └── NO  ──► Fallback: Sample y ~ Ps (from student)
  1. Reverse Speculative Decoding (RSD): Unlike standard speculative decoding (where a small model proposes and a large model verifies), in RSD the teacher LLM proposes a candidate token, and the student LLM decides whether to accept it based on its own distribution ($P_s(y) \ge p_{th}$). If the token falls below the threshold $p_{th}$, generation falls back to sampling directly from the student.
  2. Rejection Sampling: The system generates multiple candidate traces per problem and selects logically correct traces that reach the right answer.
  3. Hybrid Training Scheme:
  4. Primary RSD Training: Complete, correct traces are trained via standard SFT.
  5. Unsupervised Prefix Fine-Tuning (UPFT) for Unsolved Problems: If rejection sampling cannot find a fully correct solution, the first 128 tokens of an unsuccessful RSD trace are kept. This preserves the student's ability to learn initial problem framing without reinforcing incorrect conclusions.

Datasets

  • s1K / s1K-1.1: 1,000 challenging problems across mathematics, science, and logic. s1K-1.1 contains original traces distilled from DeepSeek-R1.
  • OpenThoughts3-1.2M (1k subset): A size-controlled sample of 1,000 math and science problems used to confirm cross-dataset robustness.
  • Evaluation Benchmarks:
  • AIME 2024 & AIME 2025 (Competition mathematics, evaluated via avg@64)
  • GPQA Diamond (Graduate-level scientific reasoning, evaluated via avg@8)
  • MATH500 (Challenging multi-discipline math, evaluated via pass@1)
  • MMLU-Pro (Selected): 4,849 questions requiring logical deduction (Law, Economics, Health, Psychology, Business, Philosophy).

Key Insights and Ideas

  • Sub-1% Tokens are Transfer Bottlenecks: In s1K-1.1 traces, 6.70% of tokens had $<1\%$ probability under the 0.6B student's distribution (often logical branching conjunctions and transition keywords). RSD with $p_{th} = 1\%$ reduced this ratio to 0.09%. This sub-1% metric was the single strongest predictor of distillation failure.
  • Alignment is Architectural, Not Universal: RSD traces are not universally accessible. Traces tailored for Qwen3-0.6B degraded or failed to help Gemma-3-1B, Llama-3.2-1B, Phi-4-Mini, or even larger models in the same family (Qwen3-1.7B). "Cognitive load" in language models is fundamentally architectural—each model requires traces aligned to its specific pre-trained representations.
  • Trace Quality Over Compute Volume: In compute-matched experiments, giving student self-distillation 203 rejection attempts (vs. 16 for RSD) solved more problems during generation (189 vs. 180), yet still degraded downstream accuracy (25.94% vs. 27.21% for RSD).
  • RSD Targets Sub-4B Models: Qwen3-4B benefited from raw teacher distillation directly, confirming that RSD is specifically needed when student models lack the internal capacity to handle high-surprisal reasoning paths.

Key Results

  • Qwen3-0.6B Reasoning Benchmarks:
  • Base Model: 25.95% average accuracy across AIME24, AIME25, GPQA Diamond, and MATH500.
  • Direct Distillation (s1K-1.1): Dropped to 20.64% (a 20.5% relative degradation).
  • RSD ($p_{th} = 1\%$): Improved to 27.21% (+1.26% absolute / +4.9% relative over the base model).
  • OpenThoughts3-1.2M Validation: Direct distillation dropped performance to 21.15%, whereas RSD recovered accuracy to 25.23% and achieved the highest score on AIME24 (4.01% vs. 2.71% base).
  • General Domain (MMLU-Pro): Base accuracy of 30.93% dropped to 30.54% under direct distillation, but increased to 31.39% with RSD.

Limitations and Future Work

  • Tokenizer Compatibility Constraint: RSD requires simultaneous token-level probability scoring, restricting teacher-student pairs to models sharing identical or carefully mapped tokenizers (e.g., s1.1-7B to Qwen3-0.6B).
  • Failure of Multi-Step/Iterative Training: Running RSD iteratively across multiple training rounds caused severe degradation (dropping average accuracy to 23.70%), illustrating model collapse and distribution narrowing when recursively fine-tuning on self-aligned data.
  • Sensitivity to Pre-trained Style: When applied to models with inherently concise styles (e.g., Llama-3.2-1B-Instruct, which produced traces ~4× shorter than Qwen3-0.6B), RSD yielded negligible gains (+0.09%), showing that RSD cannot easily compensate for missing pre-trained reasoning priors.

Summary updated

Abstract

Transferring reasoning capabilities from larger language models to smaller ones through supervised fine-tuning often fails counterintuitively, with performance degrading despite access to high-quality teacher demonstrations. We identify that this failure stems from distributional misalignment: reasoning traces from larger models contain tokens that are low probability under the student's distribution, exceeding the internal representation capacity of smaller architectures and creating learning barriers rather than helpful guidance. We propose Reverse Speculative Decoding (RSD), a mechanism for generating student-friendly reasoning traces in which the teacher model proposes candidate tokens but the student model determines acceptance based on its own probability distributions, filtering low probability tokens. When applied to Qwen3-0.6B, direct distillation of s1K-1.1 reasoning trace data degrades average performance across major reasoning benchmarks by 20.5\%, while the same model trained on RSD-generated reasoning traces achieves meaningful improvements of 4.9\%. Our analysis reveals that low probability tokens constitute the critical bottleneck in reasoning ability transfer. However, cross-model experiments demonstrate that RSD traces are model-specific rather than universally applicable, indicating that distributional alignment must be tailored for each student architecture's unique internal representation. Code and datasets are available at https://anonymous.4open.science/r/rsd.
BibTeX
@misc{kim2025their,
  author = {Jaehoon Kim and Kwangwook Seo and Dongha Lee},
  title = {{In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners}},
  year = {2025},
  eprint = {2509.22230},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22230},
}
Figure