TL;DR
Directly fine-tuning small language models on long, complex reasoning traces from massive teachers often backfires, degrading the student's performance. The authors introduce Reverse Speculative Decoding (RSD), where a teacher proposes reasoning steps but the student accepts them only if they fall within its own probability distribution. By filtering out high-surprisal tokens, RSD turns teacher reasoning into student-friendly demonstrations, transforming a 20.5% performance drop into a 4.9% gain on Qwen3-0.6B across standard reasoning benchmarks.
What Problem Does This Paper Address?
Supervised fine-tuning (SFT) on reasoning traces (like DeepSeek-R1 or s1K) has become the standard method for transferring problem-solving capabilities from large teachers to smaller models. While models at the 32B scale benefit dramatically, applying these exact traces to sub-4B models often paradoxically causes severe performance degradation.
The authors attribute this breakdown to distributional misalignment:
- Reasoning traces generated by massive models contain logical leaps and vocabulary transitions that have near-zero probability under the student’s internal distribution.
- Forcing a tiny model to recite these "high-surprisal" tokens overwhelms its capacity, creating learning obstacles rather than instructive guidance.
Methods
To smooth the cognitive stride of reasoning traces, the authors propose Reverse Speculative Decoding (RSD) and a hybrid distillation pipeline:
Teacher proposes token y ~ Pt
│
▼
Is Ps(y) ≥ p_th?
├── YES ──► Accept teacher token y
└── NO ──► Fallback: Sample y ~ Ps (from student)
- Reverse Speculative Decoding (RSD): Unlike standard speculative decoding (where a small model proposes and a large model verifies), in RSD the teacher LLM proposes a candidate token, and the student LLM decides whether to accept it based on its own distribution ($P_s(y) \ge p_{th}$). If the token falls below the threshold $p_{th}$, generation falls back to sampling directly from the student.
- Rejection Sampling: The system generates multiple candidate traces per problem and selects logically correct traces that reach the right answer.
- Hybrid Training Scheme:
- Primary RSD Training: Complete, correct traces are trained via standard SFT.
- Unsupervised Prefix Fine-Tuning (UPFT) for Unsolved Problems: If rejection sampling cannot find a fully correct solution, the first 128 tokens of an unsuccessful RSD trace are kept. This preserves the student's ability to learn initial problem framing without reinforcing incorrect conclusions.
Datasets
- s1K / s1K-1.1: 1,000 challenging problems across mathematics, science, and logic. s1K-1.1 contains original traces distilled from DeepSeek-R1.
- OpenThoughts3-1.2M (1k subset): A size-controlled sample of 1,000 math and science problems used to confirm cross-dataset robustness.
- Evaluation Benchmarks:
- AIME 2024 & AIME 2025 (Competition mathematics, evaluated via
avg@64) - GPQA Diamond (Graduate-level scientific reasoning, evaluated via
avg@8) - MATH500 (Challenging multi-discipline math, evaluated via
pass@1) - MMLU-Pro (Selected): 4,849 questions requiring logical deduction (Law, Economics, Health, Psychology, Business, Philosophy).
Key Insights and Ideas
- Sub-1% Tokens are Transfer Bottlenecks: In s1K-1.1 traces, 6.70% of tokens had $<1\%$ probability under the 0.6B student's distribution (often logical branching conjunctions and transition keywords). RSD with $p_{th} = 1\%$ reduced this ratio to 0.09%. This sub-1% metric was the single strongest predictor of distillation failure.
- Alignment is Architectural, Not Universal: RSD traces are not universally accessible. Traces tailored for Qwen3-0.6B degraded or failed to help Gemma-3-1B, Llama-3.2-1B, Phi-4-Mini, or even larger models in the same family (Qwen3-1.7B). "Cognitive load" in language models is fundamentally architectural—each model requires traces aligned to its specific pre-trained representations.
- Trace Quality Over Compute Volume: In compute-matched experiments, giving student self-distillation 203 rejection attempts (vs. 16 for RSD) solved more problems during generation (189 vs. 180), yet still degraded downstream accuracy (25.94% vs. 27.21% for RSD).
- RSD Targets Sub-4B Models: Qwen3-4B benefited from raw teacher distillation directly, confirming that RSD is specifically needed when student models lack the internal capacity to handle high-surprisal reasoning paths.
Key Results
- Qwen3-0.6B Reasoning Benchmarks:
- Base Model: 25.95% average accuracy across AIME24, AIME25, GPQA Diamond, and MATH500.
- Direct Distillation (s1K-1.1): Dropped to 20.64% (a 20.5% relative degradation).
- RSD ($p_{th} = 1\%$): Improved to 27.21% (+1.26% absolute / +4.9% relative over the base model).
- OpenThoughts3-1.2M Validation: Direct distillation dropped performance to 21.15%, whereas RSD recovered accuracy to 25.23% and achieved the highest score on AIME24 (4.01% vs. 2.71% base).
- General Domain (MMLU-Pro): Base accuracy of 30.93% dropped to 30.54% under direct distillation, but increased to 31.39% with RSD.
Limitations and Future Work
- Tokenizer Compatibility Constraint: RSD requires simultaneous token-level probability scoring, restricting teacher-student pairs to models sharing identical or carefully mapped tokenizers (e.g., s1.1-7B to Qwen3-0.6B).
- Failure of Multi-Step/Iterative Training: Running RSD iteratively across multiple training rounds caused severe degradation (dropping average accuracy to 23.70%), illustrating model collapse and distribution narrowing when recursively fine-tuning on self-aligned data.
- Sensitivity to Pre-trained Style: When applied to models with inherently concise styles (e.g., Llama-3.2-1B-Instruct, which produced traces ~4× shorter than Qwen3-0.6B), RSD yielded negligible gains (+0.09%), showing that RSD cannot easily compensate for missing pre-trained reasoning priors.
Summary updated