Aligning Draft and Target Distributions in Speculative Decoding

174 concepts130 papers1 proposal

Autoregressive generation in Large Language Models (LLMs) is notoriously bottlenecked by memory bandwidth. Because every single token requires loading the model's massive weight matrix into hardware caches, standard decoding fails to exploit the parallel processing power of modern GPUs. Speculative decoding breaks this bottleneck using a "draft-then-verify" paradigm: a small, lightweight "draft" model quickly generates a sequence of candidate tokens, which a large "target" model verifies in parallel during a single forward pass. As Heming Xia et al. explain in their comprehensive survey, Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding, this approach can yield massive wall-clock speedups without sacrificing output quality.

However, the entire paradigm hinges on a critical assumption: the draft model must accurately anticipate what the target model wants to say. When the probability distributions of the draft and target models diverge—a phenomenon known as draft-target distribution misalignment—the target model rejects the proposed tokens.

The Cost of Rejection

Why does misalignment matter so much? In a rigorous theoretical analysis, Ming Yin demonstrates in A Theoretical Perspective for Speculative Decoding Algorithm that the total inference time in speculative architectures is dominated by the frequency of rejections. This frequency is mathematically determined by the cumulative Total Variation distance between the draft and target distributions.

When a token is rejected during verification, the system must discard the remainder of the pre-computed block and fall back to standard sequential decoding. As Jiesong Liu et al. point out in A Drop-In Solution for On-the-Fly Adaptation of Speculative Decoding in Large Language Models, if the draft model's prediction accuracy drops too low, the computational overhead of drafting and rejection can actually make speculative decoding slower than standard autoregressive generation.

Why Draft and Target Models Diverge

To fix distribution misalignment, we must first understand why it occurs. The literature identifies three primary drivers:

  • The Intelligence Gap: Smaller models inherently lack the capacity to mimic the complex reasoning and nuanced token distributions of massive target models. Furthermore, Ofir Zafrir et al. note in FastDraft: How to Train Your Draft that standard training on human text fails to capture the unique quirks, biases, or watermarks of a specific target model.
  • Structural and Task Mismatch: Draft models are typically trained using standard cross-entropy loss for single-token prediction. However, during speculative decoding, they are deployed to generate complex tree structures or multi-token blocks. Shijing Hu et al. identify this as a "policy misalignment" in Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding, where the draft model proposes branches that the target model frequently rejects because the training objective never accounted for tree-based generation.
  • Stochastic Sampling Variance: In high-entropy or creative generation tasks, the target model uses stochastic sampling rather than greedy decoding. As Tianle Cai et al. observe in Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, high-temperature sampling increases the variance of token selection. This makes it highly improbable for a static draft model to match the exact token chosen by the target model, even if the draft's suggestion is semantically valid.

Bridging the Gap: Alignment Strategies

To unlock the true speed limits of speculative decoding, researchers have developed several distinct strategies to align draft and target distributions.

Target-Aware Distillation and Pruning

Rather than training draft models in isolation, modern techniques distill the target model's distribution directly into the drafter. In Faster Speculative Decoding via Effective Draft Decoder with Pruned Candidate Tree, Huanran Zheng leverages the target LLM's internal hidden states to guide the draft model's predictions.

To maximize the limited capacity of tiny drafters, Yuezhou Hu et al. introduce AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders, which filters out "hard-to-learn" tokens during distillation. This allows the draft model to focus exclusively on tokens it can realistically master, maximizing token acceptance rates within a strict parameter budget.

Online and Adaptive Drafting

Static draft models inevitably suffer from distribution shifts when real-world user queries deviate from their training data. To counter this, Xiaoxuan Liu et al. propose Online Speculative Decoding, which continuously fine-tunes the draft model in real-time using the target model's corrections.

Similarly, Ramchalam Kinattinkara Ramakrishnan et al. develop OmniDraft: A cross-vocabulary, online adaptive drafter for on-device speculative decoding, allowing a single drafter to dynamically adapt to user data on the fly, maintaining high acceptance rates across shifting contexts.

Relaxed and Contextual Verification

A powerful alternative to perfect distribution matching is changing the rules of verification. Instead of requiring exact token identity, Cheng Ding et al. introduce Consultant Decoding: Yet Another Synergistic Mechanism, which replaces rigid distribution matching with a flexible likelihood threshold.

Taking this further, Yuming Du proposes Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment, training a tiny "judge" layer to accept draft tokens based on their contextual and semantic correctness rather than strict probability matching. In sampling-heavy settings, Ying Li et al. implement a risk-bounded relaxed acceptance rule in ARC-Decode: Accelerated Decoding with Risk-Bounded Acceptance to absorb stochastic variance without degrading output quality.

Architectural Smoothing

Sometimes, the distributional gap between a tiny draft model and a massive target model is simply too wide to bridge directly. Sanghyun Byun et al. address this in 3-Model Speculative Decoding by inserting an intermediate "qualifier" model to smooth the transition between the draft and target distributions.

Alternatively, the framework proposed by Tianle Cai et al. eliminates the separate draft model entirely. By appending multiple parallel decoding heads directly to the target model's backbone, they ensure the draft generation shares the exact same underlying representations as the verifier, bypassing the draft-target alignment problem altogether.

Key Foundational Work

DistillSpec: Improving Speculative Decoding via Knowledge Distillation (2023) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 121 papers
  2. Version 1 · 2026-10-04 · 190 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_draft_target_distribution_misalignment_7d607532,
  title = {draft-target distribution misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/draft-target-distribution-misalignment/rdr_7d6075326d214ff3b6e012d4786514a7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_draft_target_distribution_misalignment_7d607532_v2,
  title = {draft-target distribution misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/draft-target-distribution-misalignment/rdr_7d6075326d214ff3b6e012d4786514a7/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake