Target Regeneration Shrinks the Gains of Draft-Training Objectives in Speculative Decoding

Jungseob Lee, Hogun Seong, Euijun Choi, Sugyeong EoarXiv 2026 · arXiv

TL;DR

Advanced training objectives for speculative decoding, like knowledge distillation and survival gating, show impressive gains on standard "external" datasets. However, this paper reveals that these gains largely evaporate when the training data is regenerated by the target model itself—a common practice in production pipelines. The authors argue that researchers should measure new objectives against a target-regenerated baseline to avoid overstating their real-world impact.

What Problem Does This Paper Address?

Speculative decoding is a popular technique to speed up Large Language Model (LLM) inference. It uses a small, fast "draft" model to predict a sequence of tokens, which a large "target" model then verifies in parallel. The speedup depends on the acceptance rate: how many tokens the target model agrees with.

To maximize this rate, researchers develop specialized training objectives for draft models. Usually, these objectives are benchmarked on "external" datasets (e.g., responses written by GPT-4). However, many state-of-the-art pipelines use target regeneration, where the target model rewrites the training corpus to match its own output distribution.

The authors identify a critical gap: we don't know if the fancy training objectives that look great on external data actually provide any benefit when the training data is already perfectly aligned with the target model through regeneration.

Methods

The researchers compared three primary training objectives for draft models:

  1. Cross-Entropy (CE): Standard training on the tokens present in the corpus.
  2. Knowledge Distillation (KD): Training the drafter to match the target model's full probability distribution for each token.
  3. KD + Gate: Distillation, but only applied to positions that would survive the target's verification (i.e., ignoring "dead-end" paths in the training data).

They tested these objectives across four versions of the same training data:

  • External: Original corpus responses.
  • Full Regeneration: The target model's greedy responses to the same prompts.
  • Truncated / Loss Window: Regenerated responses matched to the length of the original external responses to control for the fact that regenerated text is often longer.
Figure 1: One training example with external (a) and target-regenerated (b) responses. Below each corpus token is the target’s greedy token, and red marks dis- agreement. Cross-entropy (CE) trains the outlined slot on its corpus token, distillation (KD) on the target’s dis- tribution.

Datasets

The study utilized several large-scale datasets for both vision-language and text-only tasks:

  • Vision-Language: Subsets of ALLaVA and ShareGPT4V. Evaluation was performed on COCO, TextVQA, and DocVQA.
  • Text-Only: A subset of UltraChat 200k. Evaluation covered a wide range of benchmarks including MT-Bench, Alpaca, GSM8K, AIME, HumanEval, and LiveCodeBench.

Key Insights and Ideas

  • The "Agreement" Trap: On external data, the corpus tokens only match the target's greedy predictions about 58-65% of the time. Regeneration pushes this "agreement" to ~95%. Because the data is already so well-aligned, simple Cross-Entropy becomes a much stronger baseline, "shrinking" the relative gains of KD and gating.
  • Greedy Labels are Most of the Gain: On external data, simply replacing the original corpus tokens with the target's greedy tokens (without full distillation) accounts for the majority of KD's reported benefits.
  • Baseline Shift: Regeneration itself provides a massive boost to performance. A draft model trained with basic CE on regenerated data often outperforms a model trained with advanced KD on external data.
  • Shrinkage is Universal: This "gain shrinkage" persists across different model architectures (parallel vs. autoregressive drafters), training lengths, and hyperparameters.
Figure 5: Objective gains when a share of the external ALLaVA responses is replaced by the target’s greedy regenerations, with 95% intervals.

Key Results

The main metric used is Mean Accepted Tokens (MAT)—the average number of tokens accepted per speculative round.

  • KD Gains Shrink: On the ALLaVA dataset, KD provided a 14.4% improvement over CE on external responses. After full regeneration, that gain dropped to just 3.7%.
  • Gating Gains Vanish: The "survival gate" objective added 4.9% on external data but showed no measurable gain (-0.5%) after regeneration.
  • Training Length Matters: The authors found that CE performance often degrades with longer training on external data, while KD remains stable. This makes KD look better at long training durations, but this effect is also reduced after regeneration.
Figure 4: (a) Vision MAT of CE and KD on ALLaVA by training length. (b) KD gain over CE with 95% prompt bootstrap intervals. Every point is a separately trained and annealed schedule, not a checkpoint of a longer run.

Limitations and Future Work

  • Compute Constraints: The study used fixed subsets of training corpora (26k examples) due to compute limits. Results might vary on massive-scale datasets.
  • Model Specificity: While they tested both vision and text models, they focused on the Qwen3 family. Other model architectures might exhibit different sensitivities to regeneration.
  • Greedy Focus: Most evaluations were done with greedy decoding. While they touched on sampling, more work is needed to see how these dynamics change in high-temperature sampling scenarios.
  • Future Direction: The authors suggest that instead of inventing new losses for fixed corpora, researchers should focus on objectives that provide value on top of regenerated data, such as better ranking of non-greedy candidates in the speculative tree.

Summary updated

Abstract

Speculative decoding accelerates large language models by verifying drafted tokens in parallel. New draft-training objectives are usually credited with the acceptance they add over cross-entropy on externally written responses, yet many pipelines let the target rewrite them. We ask how much of that gain survives. With a vision-language target and a block drafter, we train three objectives on four versions of one corpus: cross-entropy on corpus tokens, token-level distillation from the target, and distillation gated to positions that would survive verification. Distillation and the gate add large gains on external responses, but after greedy regeneration, distillation adds a smaller gain and the gate little or none. The shrinkage persists across data orders, position weights, verifiers, decoding temperatures and, for distillation, initializations and learning rates. Most of it remains at matched supervised length and at each objective's best measured training length. A second vision corpus shows the shrinkage, as does a text-only replication, for distillation also with an autoregressive drafter. Objective gains should therefore be measured against a target-regenerated baseline and across training lengths. Our code is available at https://github.com/js-lee-AI/TargetRegeneration.
BibTeX
@misc{lee2026target,
  author = {Jungseob Lee and Hogun Seong and Euijun Choi and Sugyeong Eo},
  title = {{Target Regeneration Shrinks the Gains of Draft-Training Objectives in Speculative Decoding}},
  year = {2026},
  eprint = {2610.29013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.29013},
}
Figure