Effective Teacher Support for Reasoning Distillation

Lavinia Ghita, Dhruv DesaiarXiv 2026 · arXiv

TL;DR

Knowledge distillation for LLMs typically requires matching a teacher's full vocabulary distribution, which is computationally expensive and memory-intensive. This paper proves that for reasoning tasks, you can discard 99.99% of the teacher's vocabulary—retaining as few as 15 tokens—without any observed degradation in benchmark performance, while slashing training time by 38% and peak memory by 24%.

What Problem Does This Paper Address?

Knowledge Distillation (KD) is the process of training a smaller "student" model to mimic the predictive distribution of a larger "teacher" model. In the world of Large Language Models (LLMs), this usually involves matching the teacher's softmax distribution over the entire vocabulary (often 150,000+ tokens).

This creates two major bottlenecks:

  1. Storage: Caching full-vocabulary targets for "offline" distillation is storage-prohibitive (terabytes of data).
  2. Compute/Memory: "Online" distillation requires running the massive teacher model alongside the student, and materializing huge logit matrices for the loss calculation.

The authors ask a fundamental question: How much of the teacher's "support" (the vocabulary tail) do we actually need? If we only keep the top few tokens, does the student lose its ability to reason?

Methods

The researchers employed a Rank-$k$ Truncated Forward KL approach. Instead of calculating the loss over the full vocabulary $V$, they only consider the top $k$ most probable tokens from the teacher.

Key technical components include:

  • Renormalization: They re-normalize the probabilities of the retained $k$ tokens so they sum to 1. This "renorm" objective is crucial because it allows the student to ignore the rest of the vocabulary during the backward pass, saving memory.
  • Support Construction: They tested a "ladder" of support sizes, sweeping from 100% of the vocabulary down to just 0.01% ($k=15$ for Qwen3).
  • Structural Safeguards: For models like Qwen3 that use specific tags (e.g., <think> and </think>), they explicitly ensured these tokens were included in the top-$k$ set to prevent the student from "forgetting" how to structure its reasoning traces.
  • Systems Optimization: By caching only 15 tokens per position, they removed the teacher from the training loop entirely and used a "compact $k$-row" output computation to avoid materializing full-vocabulary student logits.

Datasets

  • Training: The primary experiments used the Nemotron Post-Training Dataset v2 (specifically the reasoning-on subset of ~545k chat records). They also conducted an "answer-only" experiment using the Puzzle-KD dataset.
  • Evaluation: The models were primarily evaluated on MMLU-Pro (a robust reasoning-heavy benchmark) and GPQA-Diamond (expert-level science questions).

Key Insights and Ideas

  • The Tail is Negotiable: Even though the teacher's distribution has a long tail, Figure 1 shows that teacher mass is highly concentrated. For Qwen3, just 15 tokens capture >99% of the probability mass in over 93% of supervised positions.
Figure 1: Effective teacher support on reasoning traces. Empirical CDFs of 𝑘𝜌over the 905,193,844 supervised positions remaining after excluding the first supervised position in each example, for which 𝑘𝜌= 1. The vertical line marks 𝑘= 15.
  • Disagreement $\neq$ Degradation: As $k$ decreases, the student's distribution naturally starts to "disagree" more with the teacher's full distribution (as seen in the rising KL diagnostic in Figure 4). However, this disagreement does not translate to lower benchmark scores. For greedy decoding, the student only needs to get the "top" of the distribution right.
Figure 4: Full-vocabulary KL across retained support size. Three-seed in-sample diagnostic on the same 256 training examples (430,657 supervised positions). The line shows the mean, light points show individual seeds, and the dashed line marks the full-𝑉mean.
  • Model-Family Sensitivity: The "safe" cutoff is not a universal constant. While Qwen3 was stable at 0.01% support, the Nemotron Nano model required 0.1% support to avoid failing to emit reasoning termination tags.

Key Results

  • Zero Performance Loss: Distilling Qwen3 4B $\rightarrow$ 1.7B at $k=15$ (0.01% support) yielded an MMLU-Pro score of 0.477, statistically indistinguishable from the 0.474 achieved with the full vocabulary.
  • Massive Efficiency Gains:
    • Step Time: Caching $k=15$ targets reduced the mean training step time by 38% by removing the teacher forward pass.
  • Memory: The compact $k$-row head reduced peak allocated memory by 24% compared to caching the full vocabulary, and by 46% compared to live-teacher training.
  • Consistency: These findings held true across different model sizes (Qwen3 8B $\rightarrow$ 4B) and even in answer-only training settings.
Figure 2: Generation across retained support size. Three-seed MMLU-Pro (left) and GPQA-Diamond (right). Lines and large markers show means; light points show individual seeds. Horizontal lines mark the full-vocabulary student and frozen Qwen3-1.7B. Both panels use a log-scaled support axis but different vertical ranges.

Limitations and Future Work

  • Decoding Strategy: The study focused on greedy decoding. The authors note that if a student is intended for high-temperature sampling (where the "tail" is actually sampled), a larger $k$ might be necessary.
  • Failure Boundaries: While they identified that Nemotron fails below 0.1%, they did not precisely locate the failure boundary for Qwen3, which remained stable even at the lowest tested support (0.01%).
  • Task Specificity: The results are scoped to reasoning and chat datasets; it remains to be seen if more diverse or creative writing tasks require more of the teacher's distribution tail.

Summary updated

Abstract

Forward-KL language-model distillation normally computes dense teacher and student scores over the full vocabulary at every supervised position, making training and stored teacher targets expensive. We instead distill from the teacher's highest-probability vocabulary fraction, renormalizing the retained mass as we sweep support from 100% to 0.01%. Our main study distills Qwen3 4B -> 1.7B. We confirm the broader support-reduction finding on larger Qwen3 8B -> 4B and Nemotron Nano 12B -> 9B pairs and under answer-only supervision. We evaluate generation primarily on MMLU-Pro and secondarily on GPQA-Diamond. Across the Qwen3 settings, retaining 0.01% of the vocabulary causes no observed MMLU-Pro degradation relative to full-vocabulary distillation, although disagreement with the complete teacher distribution increases. Nemotron Nano matches full-vocabulary MMLU-Pro at 0.1%, but smaller fractions fail to emit the closing reasoning tag, showing that the reliable cutoff depends on the model family. At 0.01% support, caching the retained next-token scores from the teacher forward pass cuts training step time by 38%, while restricting student output computation to those tokens cuts peak memory by 24%; neither causes observed MMLU-Pro degradation.
BibTeX
@misc{ghita2026effective,
  author = {Lavinia Ghita and Dhruv Desai},
  title = {{Effective Teacher Support for Reasoning Distillation}},
  year = {2026},
  eprint = {2611.14850},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14850},
}
Figure