Speculative Decoding for Autoregressive Models

1,502 concepts755 papers13 proposals

Autoregressive models generate sequences token-by-token. To predict token $t$, the model requires all tokens from $1$ to $t-1$. This sequential dependency means that a large model with billions of parameters must load its entire weight matrix from High Bandwidth Memory (HBM) into the GPU cache just to compute a single token's probability distribution. As highlighted by Jared Fernandez et al. in Energy Considerations of Large Language Model Inference and Efficiency Optimizations, this makes inference highly memory-bound and computationally inefficient, accounting for up to 90% of the total operational energy demand for major AI providers.

Modern GPUs are designed for massive parallelism, meaning they can score a sequence of multiple tokens in nearly the same time it takes to score a single token, provided the tokens are known in advance. The core insight of speculative decoding is to exploit this asymmetry: use a cheap mechanism to guess multiple future tokens, and then use the expensive target model to verify them all in parallel in a single forward pass.

The Core Mechanism of Draft and Verify

The speculative decoding paradigm, comprehensively surveyed by Heming Xia et al. in Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding, operates via a two-stage "draft-then-verify" loop:

  1. Drafting: A lightweight, low-latency mechanism (the "drafter") quickly generates a sequence of $K$ candidate future tokens.
  2. Verification: The large target model (the "verifier") evaluates the entire block of $K+1$ tokens simultaneously.

Because the verifier processes the candidate sequence in parallel, it computes the exact conditional probabilities for all proposed positions at once. The system then determines the longest prefix of candidate tokens that matches the verifier's distribution criteria. If a candidate token is rejected, the verifier provides a corrected token for that position, and the remaining candidates are discarded.

This back-off strategy, originally explored in early forms like Blockwise Parallel Decoding for Deep Autoregressive Models by Mitchell Stern et al., ensures that the final output remains mathematically identical to standard sequential decoding. It achieves a lossless wall-clock speedup by skipping multiple sequential generation steps whenever the drafter's guesses align with the verifier's logic.

Ensuring Lossless Verification

A critical requirement of speculative decoding is that it must not degrade generation quality. For greedy decoding, verification is simple: a token is accepted if it matches the argmax of the target model's distribution. However, for stochastic sampling, the verification must be unbiased.

To achieve this, speculative sampling uses a modified rejection sampling scheme. Let $p(x)$ be the target model's probability distribution and $q(x)$ be the draft model's distribution. A candidate token $x$ is accepted with a probability of:

$$\min\left(1, \frac{p(x)}{q(x)}\right)$$

If the token is rejected, the system samples a replacement token from an adjusted distribution:

$$\max\left(0, p(x) - q(x)\right)$$

which is then normalized over the vocabulary. This ensures that the final sampled token follows the exact distribution $p(x)$ of the target model, making the acceleration completely lossless.

To further optimize this process, researchers have developed advanced coupling methods. For instance, Majid Daliri et al. in Coupling without Communication and Drafter-Invariant Speculative Decoding introduced a method using the Gumbel-Max trick to decouple the verification process from the specific proposal distribution, ensuring that the final output remains identical even if the underlying draft model is swapped or updated. Similarly, Zhengmian Hu et al. proposed Accelerated Speculative Sampling Based on Tree Monte Carlo, treating generation as a path-sampling problem to accept more tokens per step without introducing statistical bias.

Diverse Drafting Strategies

The efficiency of speculative decoding depends heavily on the draft model's "hit rate" (the percentage of accepted tokens) balanced against its computational overhead, as noted by Krishna Teja Chitty-Venkata et al. in MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models. The field has evolved several distinct paradigms for generating drafts:

Beyond Speed: Expanding the Paradigm

While speculative decoding was conceived as an inference acceleration technique, recent work demonstrates that the draft-verify framework can be extended to improve model capabilities, safety, and multi-model steering.

In Speculative Contrastive Decoding, Hongyi Yuan et al. integrate a contrastive objective directly into the verification phase. By evaluating draft tokens against a distribution derived from the difference between the expert and amateur models, they penalize common errors made by the smaller model, simultaneously accelerating inference and reducing hallucinations.

Furthermore, the framework can be used to enforce safety constraints or steer text generation. In ROOT DEFENCE STRATEGIES: ENSURING SAFETY OF LLM AT THE DECODER LEVEL, Xinyi Zeng et al. utilize the look-ahead nature of speculative prediction to perform step-by-step safety evaluations and corrections before tokens are finalized. For controlled text generation, the authors of Controlled Text Generation via Language Model Arithmetic generalize speculative sampling to evaluate complex multi-model compositions. By using a single base model to propose tokens and only invoking expensive steering arithmetic during the verification step, they achieve substantial speedups while maintaining precise control over the output.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 642 papers
  2. Version 1 · 2026-10-04 · 562 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_speculative_decoding_c8a35a73,
  title = {speculative decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/speculative-decoding/rdr_c8a35a737b0d4ff68d7894c3848cbea4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_speculative_decoding_c8a35a73_v2,
  title = {speculative decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/speculative-decoding/rdr_c8a35a737b0d4ff68d7894c3848cbea4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake