Distillation Corpora as a Byproduct of Speculative Decoding

Training a small "student" language model to imitate a large "teacher" works best when the student sees the teacher's full probability distribution over the next token, not just the token the teacher picked. This is called distributional or white-box distillation. The catch is cost. Someone has to generate the training text with the teacher, then either store a vocabulary-sized probability vector for every token (terabytes for million-sample corpora) or run the teacher again over the whole corpus during training. Pipelines that regenerate large corpora with the target model, such as the 1.4M-conversation regeneration in SpecForge, pay this cost in full and treat every sample and token the same.

This note proposes generating the distillation corpus with speculative decoding: the student drafts several tokens, and the teacher checks them all in one parallel forward pass. This choice has three consequences that existing work has not combined. Speculative sampling is lossless, so the text is distributed exactly as if the teacher had written it alone, while costing fewer sequential teacher passes. The teacher's verification pass already computes its full next-token distribution at every position of the final sequence, so the soft labels arrive at no extra cost and no recompute pass is needed. And the per-position acceptance probability measures, for free, how far the student currently is from the teacher at each token. That measurement tells us where the stored soft labels are worth their bytes.

Core Claim

At a position where the teacher's next-token distribution is $p$ and the student's is $q$, speculative sampling accepts the student's draft with probability

$$ a = \sum_{v} \min\big(p(v), q(v)\big) = 1 - \mathrm{TV}(p, q), $$

where $v$ ranges over the vocabulary and $\mathrm{TV}$ is total variation distance. This identity follows directly from the standard accept/reject rule.

Positions with $a$ near 1 add almost nothing to a divergence-based distillation loss, because the student already matches the teacher there. Positions with low $a$ carry most of the remaining signal. We claim that a corpus built this way matches the student quality of conventional offline white-box distillation (teacher regeneration followed by full logit recompute) at a fraction of the teacher compute and storage. That corpus has three properties: it is generated in a few rounds with the current student as drafter, it stores full teacher distributions only at low-acceptance positions, and it keeps the plain sampled token elsewhere. Quality is measured as drafter acceptance rate and downstream task accuracy.

Tracking acceptance across rounds adds a second, sharper claim. Positions that stay low-acceptance after the student has trained on them are the "unlearnable" tokens that AdaSPEC removes using a separately trained reference model. We can identify them without that throwaway model.

Relation to Prior Work

Four papers supply the method, and each lacks one of the pieces we need.

  • DistillSpec showed that a draft model's value comes from matching the target's distribution, and it used on-policy student generation. However, it treats all tokens equally and does not account for corpus construction cost.
  • Speculative Knowledge Distillation uses interleaved sampling: the student proposes, and the teacher replaces tokens outside its top-K.
  • SpecKD uses propose-and-verify to choose which tokens enter the loss, and finds that where the student learns matters more than the choice of divergence.

These last two run online, with the teacher in the training loop, and neither produces a reusable corpus. Neither keeps exact teacher-distributed samples either, because top-K replacement is not lossless.

Two papers frame the regime and supply its central tension. Training Domain Draft Models reports three relevant findings:

  • White-box distillation beats black-box SFT by 2–10% in acceptance rate.
  • Magpie-synthesized prompts reach 80–93% of the acceptance obtained with real user queries.
  • Offline distillation beats online distillation by 11–25%.

It also notes that dense logit storage is the bottleneck of offline white-box training. Online Speculative Decoding, by contrast, harvests verification signals at inference time but cannot ship a pre-trained student. Our round-based offline generation keeps offline training's stability and gives a partly on-policy prefix distribution, because the drafter that shapes the generation budget is the current student. The text itself remains teacher-distributed.

Two results constrain the design.

  • Sparse logits. Sparse Logits Suffice to Fail Knowledge Distillation found that zeroing non-top-$k$ logits hurt students by over 20% on CIFAR-100. Our compression therefore drops whole positions rather than truncating distributions within a position. Top-$k$ truncation is kept as a baseline, because that result is from vision and may not transfer to large vocabularies.
  • Unlearnable tokens. AdaSPEC found that tiny drafters gain over 5% acceptance on GSM8K when the hardest tokens are removed. This seems to contradict our plan of spending storage on low-acceptance positions. We address it directly: a low $a$ can mean "not yet learned" or "never learnable," and only the history across rounds can tell them apart. This is tested explicitly.

Research Program

We build a generator on top of an existing speculative sampling implementation. For every final-sequence position it records:

  • the accepted or corrected token;
  • the acceptance probability $a$;
  • the teacher distribution, which is already resident in GPU memory from the verification pass.

A storage policy then writes the full teacher distribution for positions with $a$ below a threshold $\tau$, and only the token elsewhere. $\tau$ is set to meet a fixed byte budget.

Generation proceeds in R rounds (we plan R = 3–4). Prompts come from Magpie-style self-synthesis or existing prompt sets. Round r drafts with the student trained on rounds 1..r−1. This has two effects:

  • Teacher cost per token falls as acceptance rises.
  • The stored positions shift toward what the student still gets wrong.

The training loss is a divergence (forward KL for sampled decoding, following DistillSpec's analysis) at stored positions, plus cross-entropy on the sampled token elsewhere.

A persistence score marks positions as unlearnable: positions whose prompt-prefix pattern stays below $\tau$ across rounds. These positions are down-weighted, which is our reference-model-free version of AdaSPEC's selection.

Using a teacher and student that share a tokenizer (for example Qwen2.5-7B-Instruct teaching Qwen2.5-0.5B) avoids the sequence-alignment machinery that Specializing Smaller Language Models towards Multi-Step Reasoning needed. A second pair at a larger capacity gap tests whether the method still holds there.

Baselines

Naive reference: black-box SFT. The student is fine-tuned with cross-entropy on teacher-generated text produced by ordinary teacher sampling over the same prompts. This represents doing nothing distributional. It is also the cheapest way to build the corpus apart from our method, so we report its teacher GPU-hours alongside ours.

Strongest offline competitor: dense white-box distillation. This is the SpecForge recipe (SpecForge): regenerate all responses with the teacher, then distill on full teacher distributions recomputed during training. It is the method our approach must match in quality while beating it on cost.

Strongest selective method: AdaSPEC. It trains a reference drafter first, then distills only the 40% most learnable tokens. Its token selection is the closest prior idea to ours, so we run it on the same generated corpus.

Strongest online method: Speculative Knowledge Distillation's interleaved sampling. It removes stored corpora entirely, so it is the natural alternative to anything offline.

Fairness rules apply to every baseline. All methods use the same prompt set, student initialization, training steps, and hyperparameter search budget (an identical grid over learning rate and loss temperature). Each method's total teacher GPU-hours (generation plus any recompute) and bytes stored are logged.

Reported numbers to beat on the shared benchmark:

  • AdaSPEC reports an acceptance-rate gain of "over 5%" on GSM8K for Pythia 31M→1.4B.
  • SpecKD reports GSM8K Pass@1 rising from 76.27 to 77.82 on Qwen2-Math-1.5B.
  • DistillSpec reports 10.7x latency reduction on GSM8K with T5-Small/T5-XL.

We found no reported result on Lacuna for the teacher compute cost of SpecForge's regeneration step. We will measure that cost ourselves.

Win condition. Our method wins if it reaches at least the dense white-box acceptance rate and task accuracy, within run-to-run noise over three seeds, using at most half the teacher GPU-hours and at most a tenth of the logit storage. The halving and tenth targets are our pre-registered bar, not prior numbers. Our method wins outright if it also beats AdaSPEC on acceptance at the smallest student size.

Ablations. Three ablations isolate the key ingredients:

  • storing the free soft labels at every position (no gating), which separates the effect of free labels from the effect of selection;
  • random position gating at the same byte budget, which tests whether acceptance-based selection is what matters;
  • a single round versus R rounds, which tests whether drafting with the improving student matters.

Why not just…?

Why not just do SFT on teacher outputs and skip soft labels altogether? The domain-draft study reports white-box distillation beating black-box SFT by 2–10% in acceptance (Training Domain Draft Models). Recurrent drafters likewise benefit from full distributions (Recurrent Drafter). Our point is that once generation runs speculatively, soft labels cost no extra teacher compute, only storage, and gating bounds the storage. The SFT baseline, measured at equal teacher GPU-hours, settles whether the stored labels earn their bytes.

Why not just store top-$k$ logits at every position? This is a real threat. The negative evidence (Sparse Logits Suffice to Fail Knowledge Distillation) comes from image classification, and LLM next-token distributions are often very peaked. We therefore run top-$k$ storage at matched bytes, with and without renormalization. We also test the hybrid: top-$k$ at gated positions. If uniform top-$k$ ties with position gating, the honest conclusion is that the free soft labels and the cheaper generation are the contribution, and the gating is not.

Why not just distill online with interleaved sampling or Online Speculative Decoding and never build a corpus? The online approach gives up three things: the corpus is not reusable across students and epochs, a pre-deployed student requires the teacher to stay in the training loop, and offline training was measured 11–25% better in acceptance in the domain-draft comparison. We run interleaved sampling at equal teacher GPU-hours. If it matches our offline student, the reusability argument is all that remains, and we will say so.

Why not just use AdaSPEC's reference-model filtering on a conventionally generated corpus? AdaSPEC needs a full extra training run and a dense teacher recompute to score tokens. Our persistence score comes from acceptance probabilities already computed during generation. The comparison with AdaSPEC, together with an ablation that replaces persistence with AdaSPEC's score, shows whether the free signal identifies the same unlearnable tokens.

Proposed Experiments

Main experiments. These cover four settings:

  • GSM8K reasoning, the shared benchmark across DistillSpec, AdaSPEC, SpecKD, and Speculative Knowledge Distillation;
  • XSum and CNN/Daily Mail summarization;
  • open-ended chat, with ShareGPT-style prompts and MT-Bench evaluation;
  • one cold-start domain, function calling with Hermes/Apigen prompts as used in the domain-draft study, where prompts are synthesized because real logs are unavailable.

Metrics. We report:

  • drafter token acceptance rate and end-to-end speedup when the student serves as drafter for the teacher;
  • student task accuracy (GSM8K Pass@1, ROUGE, MT-Bench score);
  • teacher GPU-hours to build the corpus;
  • stored bytes.

Decisive checks. Five checks decide the outcome:

  1. Pareto curve. Plot acceptance rate against teacher GPU-hours and against bytes, sweeping $\tau$ and R. The proposal stands if its curve dominates dense white-box distillation and SFT.
  2. Unlearnable-token conflict with AdaSPEC. Compare gating that keeps all low-acceptance positions against gating that also drops persistently low ones, at the smallest student. If keeping them hurts, AdaSPEC's finding is confirmed in our setting, and persistence filtering becomes required rather than optional.
  3. Free-label equivalence. The stored verification distributions must reproduce a dense teacher recompute on the same text, up to numerical precision. This confirms that the free soft labels are genuinely free.
  4. Generation savings. Teacher passes per generated token must fall across rounds as the student improves. Otherwise the multi-round design adds nothing over a single round with the initial student as drafter.
  5. Larger capacity gap. Repeat the main comparison with the second teacher-student pair, where AdaSPEC's selection matters most.