Eliminating the Draft Model in Speculative Decoding

82 concepts69 papers1 proposal

Autoregressive decoding is the primary bottleneck in large language model (LLM) inference. Because these models generate text one token at a time, every single token requires loading billions of parameters from high-bandwidth memory (HBM) to the GPU's cache (SRAM). This makes inference heavily memory-bandwidth bound rather than compute-bound; the processor spends most of its time waiting for data to arrive.

Speculative decoding emerged as a powerful solution to this bottleneck. The core intuition is simple: use a tiny, blazing-fast "draft" model to guess a sequence of $K$ future tokens, and then run the massive "target" model once to verify all $K$ tokens in parallel. Because the target model can process multiple tokens in a single forward pass just as fast as it processes one, any accepted draft tokens represent a pure speedup.

However, traditional speculative decoding introduces significant operational friction. It requires finding, training, and maintaining a separate auxiliary draft model that must share the exact same vocabulary (Timor et al.) and closely mimic the target model's probability distribution (He et al.). Co-hosting two distinct models increases memory overhead and complicates deployment pipelines (Wu et al.).

To eliminate this complexity, a major research direction focuses on self-speculative and training-free decoding. Instead of relying on an external auxiliary model, these techniques force the target model to generate its own drafts or use lightweight, non-neural heuristics to predict the future.


Self-Speculative Decoding via Early Exits

Why look outside the target model for a drafter when the model can draft for itself? Deep neural networks process information hierarchically: early layers capture rough semantic concepts and syntax, while later layers refine the exact token probabilities. Self-speculative decoding exploits this by using the early layers of the target model as the draft model.

Standard Pass:   [Layer 1] -> [Layer 2] -> ... -> [Layer N] -> Token
Early-Exit Pass: [Layer 1] -> [Layer 2] -> Draft Token (Skip remaining layers)

Elhoushi et al. demonstrate this mechanism in LayerSkip. By training the model with layer dropout and a shared early-exit loss, they enable a single set of weights to act as both drafter and verifier. The model generates a draft sequence by exiting early, and then verifies it by running the remaining layers, perfectly sharing the same memory and key-value (KV) cache.

To make this approach more efficient and avoid wasting compute on poor guesses, Liu et al. introduce Kangaroo, which adds a "double early-exiting" strategy. It exits at the layer level to generate a draft token, but also monitors token-level confidence. If the confidence of a drafted token drops below a certain threshold, the drafting phase halts immediately, preventing the system from speculating down an unlikely path.

For practitioners who want to accelerate off-the-shelf models without expensive retraining pipelines, Xia et al. propose SWIFT, an on-the-fly layer-skipping framework that requires zero additional training. Similarly, Ning et al. develop CAS-Spec, which dynamically skips intermediate layers based on the complexity of the text being generated in real-time.


Multi-Head Parallelism

Another way to eliminate the auxiliary model is to expand the target model's architecture internally. Instead of running a sequence of forward passes to generate a draft, what if the model could look ahead directly from its final hidden states?

Cai introduces this concept with Medusa. Instead of a separate model, Medusa appends multiple lightweight decoding heads to the backbone LLM's final layer.

  • Head 1 predicts the token at position $t+1$.
  • Head 2 predicts the token at position $t+2$.
  • Head 3 predicts the token at position $t+3$.
                  /-> Head 1 (Predicts t+1)
[LLM Backbone] -> ---> Head 2 (Predicts t+2)
                  \-> Head 3 (Predicts t+3)

Because these heads are simple linear layers, they add negligible computational overhead. They generate a tree of multiple candidate continuations simultaneously. The backbone model then uses a tree-based attention mechanism to verify all candidates in a single forward pass. Tuli et al. utilize a similar multi-token prediction paradigm in DynaMo, pairing it with a dynamic back-off strategy to maintain quality when confidence drops.


Training-Free Heuristics and Token Recycling

The most lightweight branch of this research direction eliminates neural network drafting entirely. Language is highly repetitive; phrases, idioms, and code structures recur frequently within the same document or prompt. We can exploit this structure using classical data structures and token manipulation.

Luo et al. introduce this intuition via Token Recycling. During speculative decoding, many generated draft tokens are rejected by the target model and thrown away. Token Recycling captures these "trash" tokens and stores them in a tiny, dynamically updated adjacency matrix. When a token is generated, the system looks up its neighbors in the matrix to instantly assemble a draft sequence for the next step without any neural network overhead.

Other methods look backward into the prompt itself. Somasundaram et al. present PLD+, which inspects internal model artifacts like attention maps to identify and recycle long sequences of text directly from the input prompt.

Taking this to its logical conclusion, Stewart et al. show in The N-Grammys that a simple, learning-free $n$-gram model built on the fly from the generation history serves as an incredibly effective speculator. For highly structured workloads like AI agents or code generation, Oliaro et al. demonstrate in SuffixDecoding that maintaining a suffix tree of previous outputs allows the system to speculate long token sequences with zero training or auxiliary model overhead.


Optimizing the Verification Tree

Once a draft is generated—whether via early layers, extra heads, or $n$-grams—the system must verify it. Maximizing the speedup requires making the verification process as adaptive as possible.

Li et al. demonstrate in EAGLE-2 that a drafter's confidence scores are highly correlated with the target model's actual acceptance rates. Instead of verifying a single static chain of tokens, they use these confidence scores to dynamically reshape a draft tree on the fly, allocating more branches to high-confidence paths.

To avoid wasting verification compute, Agrawal et al. introduce AdaEDL. It computes an entropy-based lower bound on token acceptance probabilities during the drafting phase. If the entropy rises too high (meaning the choices are becoming chaotic or highly uncertain), it dynamically truncates the draft length, ensuring the target model only verifies high-probability sequences.


Application to Non-Text Modalities

The core realization that models can act as their own speculators has recently expanded beyond text LLMs into visual and diffusion models.

In autoregressive text-to-image generation, where generating an image requires sequentially predicting thousands of visual tokens, Teng et al. introduce Speculative Jacobi Decoding (SJD). SJD bypasses traditional draft models by initializing a sequence of future tokens with a padding token and updating them in parallel using a probabilistic convergence criterion, cutting generation times in half without losing visual diversity.

For diffusion models, the sequential denoising steps present a similar bottleneck. The authors of Diffusion Models are Secretly Exchangeable uncover a mathematical property showing that diffusion increments are exchangeable. This allows them to introduce Autospeculative Decoding (ASD), which parallelizes Denoising Diffusion Probabilistic Models (DDPMs) by letting the model speculate its own future noise increments. Similarly, Wu et al. introduce FreeDraft-and-Verification (FreeDave), which leverages the inherent bidirectional nature of diffusion language models to self-verify multiple tokens in parallel, achieving up to a 2.83× speedup with zero training or auxiliary model overhead.

Key Foundational Work

Inference with Reference: Lossless Acceleration of Large Language Models (2023) — contributed 1 concept elements to this direction.
REST: Retrieval-Based Speculative Decoding (2023) — contributed 1 concept elements to this direction.
Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_speculative_decoding_complexity_1d4f420c,
  title = {speculative decoding complexity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/speculative-decoding-complexity/rdr_1d4f420c585545bbb3a647bc5ed6a6d0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_speculative_decoding_complexity_1d4f420c_v1,
  title = {speculative decoding complexity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/speculative-decoding-complexity/rdr_1d4f420c585545bbb3a647bc5ed6a6d0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake