Lossless Parallel Inference via Speculative Decoding

This direction is merged into another direction: speculative decoding. Its last version stays readable and citable here.

Version 1 of 132 papers41 conceptsfrozen 2026-10-04

Large Language Models (LLMs) are notoriously slow because they are "memory-bound." When generating text, a model must load billions of parameters from memory to produce just a single token. This process repeats sequentially: to generate token $N$, you must finish token $N-1$. Because the time taken to load the model weights far exceeds the time spent on actual computation, the GPU spends most of its time waiting for data.

Speculative decoding, pioneered by Yaniv Leviathan, Matan Kalman, and Yossi Matias in Fast Inference from Transformers via Speculative Decoding, breaks this sequential bottleneck. It allows a model to generate multiple tokens in a single parallel pass without changing the final output distribution.

The Draft and Verify Paradigm

The core intuition is that most tokens in a sentence are predictable. A tiny, inexpensive "draft" model can often guess the next few words as accurately as a massive "target" model.

In a speculative system, the process follows two steps:

  1. Drafting: A small model (or a heuristic) quickly generates a sequence of $K$ candidate tokens.
  2. Verification: The large target model processes all $K$ tokens in a single parallel forward pass.

Because modern GPUs can process a short sequence of tokens almost as fast as a single token, the verification step is "nearly free." If the target model agrees with the draft, we have generated $K$ tokens in the time it usually takes to generate one. If it disagrees, we simply discard the incorrect guesses and restart from the point of divergence.

Why It Works: Lossless Rejection Sampling

The most critical feature of speculative decoding is that it is lossless. The final text is mathematically identical to what the large model would have produced alone. This is achieved through a modified rejection sampling scheme described by Geoffrey Irving and colleagues in Accelerating Large Language Model Decoding with Speculative Sampling.

When the draft model proposes a token $x$ with probability $q(x)$ and the target model assigns it probability $p(x)$, the algorithm accepts the token with probability: $$\min(1, \frac{p(x)}{q(x)})$$ If the token is rejected, the system samples a replacement from a "residual" distribution: $\max(0, p(x) - q(x))$, normalized. This ensures that even if the draft model is stochastic, the final output perfectly mirrors the target model's probability distribution.

Eliminating the Drafter

While the original method requires two separate models, researchers have found ways to speculate using the target model itself. Edouard Grave and Armand Joulin introduced PaSS (Parallel Speculative Sampling), which uses "look-ahead embeddings" to let a single model predict future tokens. Similarly, Hanling Yi et al. developed SPACE (Smart Parallel Auto-Correct Decoding), which teaches a model to "guess" its own future outputs in a single pass, removing the need for an auxiliary drafter.

This "self-speculation" has even extended to diffusion models. Hengyuan Hu and Aniket Das demonstrated in Autospeculative Decoding that diffusion steps are "secretly exchangeable," allowing the model to speculate on future noise increments and verify them in parallel, speeding up image and robot-policy generation by up to 4x.

Optimizing the Verification Logic

Recent work has moved from verifying simple sequences to more complex structures. Xupeng Miao et al. proposed SpecInfer (Speculative Inference and Token Tree Verification), which uses a "token tree" to verify multiple potential paths at once.

Furthermore, Ziteng Sun and Uri Mendlovic proved in Block Verification Accelerates Speculative Decoding that verifying tokens one-by-one is actually suboptimal. By verifying blocks of tokens jointly, they achieved an additional 5-8% speedup for free. To handle even longer sequences, Yuxuan Zhou et al. introduced Hierarchical Speculative Decoding (Overcoming Joint Intractability), which uses a multi-level resampling strategy to maintain losslessness while accepting more tokens per cycle.

Expanding the Horizon

The speculative paradigm is now moving beyond standard text generation. Zhi-Kai Chen et al. applied these concepts to spatial contexts in Hawk (Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation), while Yuhang Xu et al. developed BubbleSpec (Turning Long-Tail Bubbles into Speculative Rollout Drafts), which uses idle GPU time during Reinforcement Learning training to pre-generate speculative drafts for future steps.

By shifting the burden of inference from sequential memory access to parallel computation, speculative decoding has become the standard for making large-scale AI models practical for real-time use.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lossless_parallel_inference_via_speculat_3b96f550,
  title = {Lossless Parallel Inference via Speculative Decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lossless-parallel-inference-via-speculative-decoding/rdr_3b96f55098f34d6a88d011b727d5f59b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_lossless_parallel_inference_via_speculat_3b96f550_v1,
  title = {Lossless Parallel Inference via Speculative Decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lossless-parallel-inference-via-speculative-decoding/rdr_3b96f55098f34d6a88d011b727d5f59b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}