Large Language Models (LLMs) are notoriously slow because they are autoregressive: to generate a sentence, they must predict each token one by one, with each step requiring a full pass through billions of parameters. Speculative Decoding (SD), introduced by Yaniv Leviathan, Matan Kalman, and Yossi Matias in Fast Inference from Transformers via Speculative Decoding, broke this bottleneck by using a tiny "draft" model to guess several tokens ahead, which a large "target" model then verifies in parallel.
However, standard speculative decoding faces a "verification wall." If the draft model is too small, its guesses are frequently wrong, forcing the target model to discard them and restart. If the draft model is large enough to be accurate, it becomes slow itself. Hierarchical Speculative Decoding (HSD) solves this by organizing multiple models into a tiered hierarchy, bridging the gap between speed and accuracy through successive refinement.
Bridging the Distributional Gap
The primary reason standard SD plateaus is the vast "distributional gap" between a tiny drafter and a massive target. A 100M parameter model simply does not think like a 70B parameter model. Sanghyun Byun, Mohanad Odema, and colleagues address this in 3-Model Speculative Decoding by introducing an intermediate "qualifier" model.
In this three-tier architecture, the tiny model generates a raw draft, the intermediate model filters and refines those tokens, and only the high-confidence sequences reach the massive target model. This ensures the most expensive compute resource only processes "high-probability" sequences. Similarly, Shensian Syu demonstrates in Hierarchical Speculative Decoding with Dynamic Window that this multi-stage verification captures a "middle zone" of tokens—those too complex for a tiny model but not requiring the full power of the target—significantly increasing the average number of tokens accepted per pass.
Recursive Drafting and the Autoregressive Bottleneck
Even a small draft model is still autoregressive, meaning it has its own "speed limit." Ziyi Chen, Xiaocong Yang, and Jie Huang propose a recursive solution in Cascade Speculative Drafting for Even Faster LLM Inference.
Instead of the draft model generating tokens from scratch, it is itself assisted by an even smaller model or a simple heuristic like n-gram retrieval. This creates a chain where each model in the hierarchy verifies the previous one and drafts for the next. By removing the autoregressive overhead from the intermediate neural models, CS Drafting achieves throughput gains that a single-assistant system cannot reach.
Self-Speculation via Internal Hierarchies
Building a hierarchy doesn't always require loading multiple separate models. Researchers have found that LLMs contain "internal hierarchies" within their own layers. Poulami Das and Sujay Sanghavi introduce HiSpec: Hierarchical Speculative Decoding for LLMs, which uses "early-exit" layers as intermediate verifiers.
By skipping layers to create a "slim-verifier," the system can correct "near-miss" predictions internally. Zhiyuan Ning and Jiawei Shao extend this in CAS-Spec, using a dynamic tree cascade to route tokens through different depths of the model based on the complexity of the text being generated. This "self-speculative" approach provides the benefits of a multi-model hierarchy without the memory overhead of storing multiple distinct parameter sets.
Hierarchical Reasoning and Semantic Lookahead
For Large Reasoning Models (LRMs) that generate long chains of thought, token-level speculation is often insufficient. Rui Pan, Zhihao Jia, and Ravi Netravali propose SpecReason, which introduces a bi-level hierarchy.
In this framework, the system speculates at two levels simultaneously:
- Thought Level: A draft model proposes entire reasoning steps or "thoughts."
- Token Level: Traditional SD accelerates the generation of those steps.
The target model acts as a "semantic critic," verifying the logic of the reasoning step rather than the exact wording. This shift from token-matching to semantic-matching, also explored by Yichao Fu and Hao Zhang in Scaling Speculative Decoding with Lookahead Reasoning, allows for multiplicative speedups by optimizing both local sequences and global reasoning trajectories.
Finding the Optimal Stack
As the number of available models and techniques grows, the challenge becomes selecting the right hierarchy. Clara Mohri and Yishay Mansour provide a theoretical foundation in Fast Inference via Hierarchical Speculative Decoding, framing the selection of a model stack as a Generalized Shortest Path problem. Their work proves that a mathematically optimal hierarchy can be found to maximize throughput, often yielding significant gains over even the best-tuned single-drafter systems.