Recovering Off-Policy Supervision for Speculative Decoding

Jungseob Lee, Chanjun Park, Sugyeong Eo, Hyeonseok MoonarXiv 2026 · arXiv

TL;DR

Speculative decoding speedups are often throttled because the small "drafter" models are trained on "off-policy" data—text generated by models other than the one they are trying to accelerate. This paper introduces a rollout-based training framework that uses Anchor-Label Relabelling (ALR) and In-Rollout Anchors (IRA) to recover lost supervision and expose drafters to the target model's own context, resulting in up to a 36.5% improvement in accepted token length without the need to pre-regenerate massive training corpora.

What Problem Does This Paper Address?

Speculative decoding uses a small, fast model (the drafter) to propose tokens that a large, slow model (the target) verifies in parallel. The efficiency of this process hinges on the "accepted length"—how many tokens the target model agrees with.

Most drafters are trained on fixed corpora (like ShareGPT or ALLaVA) generated by proprietary models like GPT-4. However, if the target model (e.g., Qwen) would have generated a different token than the corpus at any point in a block, all subsequent tokens in that training block become "off-policy." Standard training methods typically "erase" or down-weight these divergent slots to avoid confusing the drafter, leading to a massive loss of useful supervision data. While one could regenerate the entire corpus using the target model, this is computationally expensive and often legally or practically impossible if the original text must be preserved.

Methods

The authors propose two complementary components to fix this "off-policy" mismatch during the training step itself:

  1. Anchor-Label Relabelling (ALR): Instead of using the next tokens from the corpus as labels, the target model performs a short, greedy "rollout" from an anchor point in the text. The drafter is then trained to predict these target-generated tokens. This ensures that the labels always align with what the target model actually wants to say.
  2. In-Rollout Anchors (IRA): Training on correct labels isn't enough; the drafter also needs to see the kind of context it will encounter at inference time (text generated by the target model, not the corpus). IRA places secondary draft blocks directly inside the rollouts generated for ALR. By reusing the cached features from the primary rollout, IRA provides this "on-policy" context at virtually no additional computational cost to the target model.
Figure 1: Comparison of label construction methods. Erase discards divergent data, while ALR and IRA recover it through target rollouts.

Datasets

The framework was evaluated across both vision-language and text-only domains:

  • Vision-Language Training: ALLaVA (LAION split) and ShareGPT4V captions.
  • Text Training: A subset of UltraChat.
  • Evaluation Benchmarks: COCO captioning, TextVQA, and DocVQA for vision; MT-Bench, Alpaca, GSM8K, AIME24/25, HumanEval, and LiveCodeBench for text.

Key Insights and Ideas

  • Covariate Shift Matters: The "aha moment" is that the drafter's performance gap isn't just about the labels (what to predict) but the context (what it's predicting from). IRA addresses this by training the drafter on the target's own "hallucinated" continuations.
  • Efficiency through Reuse: By cleverly reusing the Key-Value (KV) cache and hidden states from the target model's rollouts, the authors get "two blocks for the price of one" (IRA), making the training significantly more efficient than full corpus regeneration.
  • One Epoch is Enough: A single epoch of ALR+IRA training can outperform the best possible "erase" schedules, proving that high-quality supervision is better than simply seeing more (but noisier) data.

Key Results

  • Accepted Length Boost: ALR + IRA improved the Mean Acceptance Length (MAT) by up to 36.5% over the DFlash baseline and 8.54% over the strong "erase" baseline on the ALLaVA dataset.
  • Decoding Speedup: The method achieved a 2.82× speedup over standard autoregressive decoding on ALLaVA, compared to 2.62× for the erase baseline.
  • Matching Regeneration: Remarkably, training on a fixed corpus with ALR+IRA matched the performance of drafters trained on a fully regenerated (and thus much more expensive) version of the same corpus.
Figure 2: MAT gain over the erase baseline across various benchmarks. ALR+IRA consistently provides the highest gains.

Limitations and Future Work

  • Greedy Rollouts: The current implementation uses greedy rollouts for training. While this works well for both greedy and sampled decoding at inference, the authors suggest that exploring rollouts that follow the sampling policy could be a natural next step.
  • Architecture Specificity: The method currently requires access to the target model's hidden states and token distributions. While standard for white-box speculative decoding, it limits application to black-box API models.
  • Future Directions: The authors plan to extend this framework to more complex drafter architectures beyond the block-diffusion style used in this study.
Figure 3: Training cost vs. performance. ALR+IRA reaches higher performance in less time than multiple epochs of the erase baseline.

Summary updated

Abstract

Block drafters for speculative decoding are commonly trained on corpora written by external models, where a single off-policy token invalidates supervision for all subsequent slots in a block. Existing approaches discard these divergent slots, resulting in severe supervision loss. To resolve this problem while preserving the training corpus, we propose a rollout-based training framework that recovers full supervision through two complementary components. The first component, Anchor-Label Relabelling (ALR), replaces corpus labels with distributions from greedy target rollouts, restoring valid supervision across all predicted slots. The second component, In-Rollout Anchors (IRA), places draft blocks directly inside these rollouts to expose the drafter to target-generated context, reusing precomputed rollout features at no additional target cost. Across fixed vision-language and text corpora, our framework increases greedy accepted length by up to 36.5% over DFlash and consistently outperforms erasing baselines. Notably, a single epoch of our method surpasses the best erase schedules. After three epochs, it matches the acceptance length of training on target-regenerated responses. These results show that our framework provides an effective and compute-efficient approach for training speculative drafters on fixed corpora without modifying the original text. Code is available at https://github.com/js-lee-AI/ALR-IRA.
BibTeX
@misc{lee2026recovering,
  author = {Jungseob Lee and Chanjun Park and Sugyeong Eo and Hyeonseok Moon},
  title = {{Recovering Off-Policy Supervision for Speculative Decoding}},
  year = {2026},
  eprint = {2609.38795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.38795},
}
Figure