Speculative Decoding for Accelerated Autoregressive Inference

1,207 concepts949 papers4 proposals

Large Language Models (LLMs) are notoriously slow during inference. This slowness is not primarily caused by a lack of raw computational power, but rather by a fundamental mismatch between how autoregressive models generate text and how modern hardware processes data.

To generate a response, an autoregressive model must produce text sequentially, token by token. Generating each individual token requires a complete forward pass through the entire network. During this forward pass, every single weight of a multi-billion parameter model must be fetched from High Bandwidth Memory (HBM) to the GPU's processor caches. Because the GPU is performing this massive memory transfer just to process a single token, the execution is entirely memory-bandwidth bound. The powerful tensor cores sit idle, waiting for weights to arrive.

Speculative decoding bypasses this memory bandwidth wall. The core intuition is simple: while generating tokens sequentially is slow, verifying multiple tokens in parallel is incredibly fast.

The Draft and Verify Paradigm

Instead of running the massive "target" model to generate every single token, speculative decoding introduces a two-stage pipeline:

  1. Drafting: A computationally inexpensive mechanism guesses a sequence of future tokens (a "draft").
  2. Verification: The large target model evaluates the entire draft sequence simultaneously in a single forward pass.
Standard Autoregressive:
[Token 1] -> Target Model -> [Token 2] -> Target Model -> [Token 3]

Speculative Decoding:
[Context] -> Cheap Drafter -> [Draft 1, Draft 2, Draft 3]
                                     |
                                     v
                       [Draft 1, Draft 2, Draft 3] -> Target Model (Parallel Pass)
                                     |
                                     v
                       [Accept,  Accept,  Reject]

Why is verifying multiple tokens in parallel faster than generating them one by one? When the target model processes $K$ draft tokens at once, it only needs to load its weights from HBM once. The GPU can parallelize the matrix multiplications across the sequence dimension, shifting the workload from being memory-bandwidth bound to compute-bound. As long as the drafting mechanism is cheap enough and its guesses are accurate enough, the wall-clock time of the entire process drops drastically.

Crucially, this process can be made completely lossless. By using modified rejection sampling algorithms, the target model can validate or reject draft tokens such that the final output distribution remains mathematically identical to standard autoregressive generation. As demonstrated by Yin et al. in A Theoretical Perspective for Speculative Decoding Algorithm, speculative decoding acts as an optimal unbiased algorithm for accelerating inference, bounded by a strict Pareto front between verification speed and draft accuracy.

Branching Out with Tree-Based Speculation

The primary bottleneck of vanilla speculative decoding is its fragility. If a draft model predicts a sequence of five tokens, and the target model rejects the very first token, the remaining four tokens are instantly wasted, even if they would have been correct in another context. Linear speculation suffers from cascading failures.

To solve this, recent innovations have shifted from linear draft sequences to tree-based speculation. Instead of guessing a single path, the drafting engine generates a tree of multiple plausible token paths.

Linear Draft:    [The] -> [cat] -> [sat] -> [on] -> [the]

Tree Draft:               /-> [cat] -> [sat]
                 [The] --+
                          \-> [dog] -> [bark]

By organizing candidates into a tree, the target model can verify multiple alternative paths simultaneously in a single forward pass using a technique known as Tree Attention. Yang et al. developed Multi-Candidate Speculative Decoding (MCSD) to exploit this parallel verification, significantly boosting token acceptance rates in scenarios where the model's confidence branches.

To maximize the efficiency of these trees, the structure should ideally adapt to the model's uncertainty. Wang et al. introduced OPT-Tree, which dynamically reshapes the draft tree at each step to maximize the mathematical expectation of the accepted length. Furthermore, the verification process itself can be optimized; Weng et al. showed in Traversal Verification for Speculative Tree Decoding that switching from traditional top-down token checks to a bottom-up sequence traversal can recover valid token sequences that standard algorithms would prematurely discard.

Designing the Ideal Drafting Engine

The performance of speculative decoding hinges entirely on the efficiency and accuracy of the drafter. Researchers have explored several distinct strategies to generate high-quality drafts without incurring heavy computational costs.

Distilled and Online Models

The most intuitive drafter is a smaller version of the target model (e.g., a 7B model drafting for a 70B model). To ensure the smaller model mimics the larger one effectively, specialized training regimes are required. Hu et al. proposed AdaSPEC, a selective knowledge distillation framework that filters out "hard-to-learn" tokens during training, allowing tiny draft models to focus their limited capacity only on tokens they can realistically master.

However, static draft models can struggle when user queries drift away from their training distribution. To combat this, Liu et al. introduced Online Speculative Decoding (OSD), which continuously fine-tunes the draft model in real-time using the target model's corrections during live inference, keeping the two models aligned.

Hardware-Efficient Architectures

Even small Transformer models carry significant memory overhead due to their Key-Value (KV) caches. To alleviate this, alternative architectures can be used as drafters. Choi demonstrated in Mamba Drafters for Speculative Decoding that State Space Models (SSMs) like Mamba can serve as highly efficient drafters. Because Mamba features linear complexity and a constant memory footprint, it eliminates the KV cache bottleneck of the drafting phase entirely.

Learning-Free and Retrieval Methods

Drafting does not strictly require a neural network. Text generation often contains highly repetitive phrases or direct copies from the input context. The authors of The N-Grammys demonstrated that a simple, learning-free $n$-gram model can serve as an effective plug-and-play speculator.

Taking retrieval a step further, Hu et al. introduced SAM Decoding, which utilizes a Suffix Automaton to rapidly retrieve draft tokens from both the current generation context and a static text corpus in $O(1)$ average time, bypassing model-based drafting overhead entirely.

Self-Drafting (Single-Model Speculation)

To avoid managing two separate models, some approaches embed the drafting mechanism directly within the target model. Liu et al. developed Speculative Decoding via Early-exiting for Faster LLM Inference, which uses the target model's own initial layers to generate draft tokens, skipping the remaining deep layers during the speculative phase. A Thompson Sampling control mechanism dynamically adjusts how many layers to skip based on recent success rates, ensuring the system adapts to the complexity of the text on the fly.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_generation_bottleneck_sequent_8897badc,
  title = {sequential generation bottleneck (sequential generation latency)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-generation-bottleneck-sequential-generation-latency/rdr_8897badc0e0f44dab0c979cbb4b2d526},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequential_generation_bottleneck_sequent_8897badc_v1,
  title = {sequential generation bottleneck (sequential generation latency)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-generation-bottleneck-sequential-generation-latency/rdr_8897badc0e0f44dab0c979cbb4b2d526/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake