Speculative Decoding for Efficient LLM Inference

This direction is merged into another direction: speculative decoding. Its last version stays readable and citable here.

Version 1 of 136 papers39 conceptsfrozen 2026-10-04

Large Language Models (LLMs) are notoriously slow at generating text because they are "memory-bound." For every single token produced, the model must load billions of parameters from memory into the processor. The actual math takes almost no time, but the data transfer creates a massive bottleneck.

Speculative decoding is a "draft-then-verify" strategy that breaks this bottleneck. Instead of using the massive, slow model to generate every token, we use a much smaller, faster "draft" model to guess a sequence of tokens. The large model then checks all those guesses in a single parallel pass. As long as the guesses are reasonably accurate, we can produce multiple tokens for the price of one large model forward pass.

The Fundamental Speed-Accuracy Trade-off

The efficiency of speculative decoding hinges on a delicate balance between two competing factors: the drafting latency and the acceptance rate.

As Heming Xia et al. explain in their comprehensive roadmap, a draft model that is too simple will have a low acceptance rate, forcing the large model to frequently discard the work and re-generate. Conversely, a draft model that is too complex—while more accurate—introduces so much computational overhead that it negates the time saved during verification.

Zhuorui Liu et al. identified a "sweet spot" for this gamble: for maximum speedup, draft models should typically be 10–20× smaller than the target model. However, Mohammadreza Pourreza et al. demonstrate in their work on S2D that this isn't a one-size-fits-all rule. Smaller target models are bottlenecked by the raw latency of the drafting process, while massive models (like Llama-3-70B) are bottlenecked by accuracy; they require "smarter" drafts to avoid the high cost of a rejected verification pass.

Improving the "Intern": Better Drafting Architectures

If we think of the draft model as a fast-typing intern and the target model as a slow, careful editor, much of the research focuses on making the intern smarter without slowing them down.

  • Recurrent Heads: Instead of independent prediction heads, Aonan Zhang introduced ReDrafter, which uses a lightweight RNN to better capture dependencies between tokens in a draft sequence.
  • Self-Speculation: Rather than training a separate model, F. Liu et al. developed Kangaroo, which uses the early layers of the target model itself to generate drafts. This "early exiting" ensures the draft model's internal representations are already aligned with the target's.
  • Feature-Level Autoregression: EAGLE improves accuracy by performing drafting at the "feature" level (the second-to-last layer) rather than just predicting tokens, which helps resolve uncertainty in the sampling process.
  • Training-Time Boosts: Feiye Huo et al. proposed RepSpec, which uses redundant linear branches during training to improve draft accuracy but merges them into a single layer for inference, providing "free" accuracy gains at runtime.

Parallelism and Hardware Utilization

Standard speculative decoding is still somewhat serial: the draft model runs, then the target model runs. Recent work seeks to overlap these stages to maximize hardware utilization.

Zhenwei Tang et al. introduced MineDraft, which parallelizes drafting and verification by maintaining two separate batches of requests. This "hides" the drafting latency entirely, allowing for the use of larger, more accurate draft models that would otherwise be too slow. Similarly, Yize Wu et al. developed EasySpec, which runs multiple draft layers simultaneously across different GPUs to eliminate idling.

Adaptive and Hierarchical Strategies

The optimal drafting strategy often changes depending on the context length or the specific hardware. Seongjin Cha et al. framed this as a resource allocation problem in KnapSpec, using a knapsack algorithm to adaptively select which layers to use for drafting as the sequence grows and attention mechanisms slow down.

For even greater efficiency, Clara Mohri et al. proposed Hierarchical Speculative Decoding (HSD). Instead of one draft model, HSD uses a stack of models of increasing size. A tiny model drafts for a medium model, which then drafts for the large target model. By finding the mathematically optimal hierarchy, they further reduce the computational cost of each "guess."

Ultimately, the goal of this research direction is to transform LLM inference from a slow, sequential process into a highly efficient, parallelized system where the cost of "thinking" is decoupled from the cost of "moving data."

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_speculative_decoding_for_efficient_llm_i_d02edb0c,
  title = {Speculative Decoding for Efficient LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/speculative-decoding-for-efficient-llm-inference/rdr_d02edb0cde65401a9982e9e43ab54964},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_speculative_decoding_for_efficient_llm_i_d02edb0c_v1,
  title = {Speculative Decoding for Efficient LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/speculative-decoding-for-efficient-llm-inference/rdr_d02edb0cde65401a9982e9e43ab54964/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}