Large Language Models (LLMs) are notoriously bottlenecked by memory bandwidth during inference. Because standard text generation happens autoregressively—one token at a time—a massive model must load all its weights from memory just to generate a single token.
To break this bottleneck, standard speculative decoding introduces a two-model paradigm, as detailed by Leviathan et al. (2023). A tiny, computationally cheap "draft" model quickly guesses a sequence of future tokens autoregressively. Then, the massive "target" model evaluates all these candidate tokens in a single parallel forward pass. If the target model accepts the draft tokens, the system generates multiple tokens for the memory cost of a single target model pass.
However, this two-model approach hits a strict performance ceiling due to a fundamental dilemma:
- If the draft model is too small, its token distribution diverges from the target model. The target model rejects its guesses, causing the system to fall back to slow serial generation.
- If the draft model is large enough to accurately mimic the target model, the draft model itself becomes slow, introducing its own severe autoregressive latency bottleneck (Chen et al. (2024)).
Hierarchical Speculative Decoding solves this problem by replacing the rigid two-model paradigm with a multi-tiered hierarchy. By cascading predictions across multiple model sizes or multiple semantic granularities, it bridges the gap between raw drafting speed and verification accuracy.
Bridging the Distributional Gap with Model-Size Hierarchies
The core intuition behind a model-size hierarchy is to treat token verification as a multi-stage filtering process rather than a single binary decision. Instead of forcing a massive target model to evaluate low-confidence guesses from a tiny draft model, intermediate models act as "qualifiers" to refine the sequence.
[ Tiny Draft Model ] ---> (Proposes raw tokens rapidly)
|
v
[ Intermediate Model ] -> (Filters low-confidence tokens / Refines draft)
|
v
[ Massive Target Model ] -> (Performs final parallel validation)
As analyzed by Wang et al. (2025), a chain of models with increasing capacity optimizes the balance between drafting speed and verification accuracy. In a three-model hierarchy, such as the one proposed by Byun et al. (2025) and explored by Syu (2025), the process unfolds in stages:
- Rapid Proposal: The smallest model generates a long sequence of tokens at minimal computational cost.
- Intermediate Filtering: An intermediate-sized model verifies these tokens using relaxed or "fuzzy" acceptance criteria. It discards obvious errors and fills in gaps, acting as a buffer that smooths the distributional transition between the tiny model and the giant model.
- Expensive Validation: The massive target model only processes the high-probability sequences that survive the intermediate filter.
By preventing the largest model from wasting cycles on poor drafts, this cascading structure significantly increases the average number of tokens accepted per target model pass. To avoid the overhead of training multiple distinct models, frameworks like the one developed by Ning et al. (2025) dynamically construct this hierarchy on the fly by skipping layers within the target model itself, adapting the cascade to the complexity of the text being generated.
Multi-Granularity Speculation: Tokens and Thoughts
The rise of Large Reasoning Models (LRMs) introduces a new bottleneck. When models use Chain-of-Thought (CoT) reasoning to solve complex problems, they generate thousands of intermediate tokens. In these scenarios, standard token-level speculative decoding fails because token-level errors accumulate rapidly over long horizons, causing early rejections.
Hierarchical speculation addresses this by shifting the unit of prediction from individual tokens to discrete logical steps. By combining high-level semantic step speculation with low-level token speculation, the system creates a multi-layered parallel execution framework (Fu et al. (2025); Shi et al. (2025)).
High-Level: [ Draft Step 1: Math Setup ] ------------> [ Draft Step 2: Calculation ]
| |
v (If Step 1 is rejected) v
Low-Level: [ Fallback Token Speculation ] [ Verify Semantic Logic ]
In this dual-layer architecture, a smaller model speculates entire reasoning paths or "thoughts" at once (Wang et al. (2025)). The large base model then acts as a "semantic critic," verifying the high-level logic rather than the exact wording (Pan et al. (2025)).
If the critic accepts the high-level trajectory, the model skips hundreds of token-level checks entirely. If a reasoning step is rejected, the system gracefully falls back to traditional token-level speculation to accelerate the regeneration of that specific segment. This ensures that local token generation and global reasoning structure are optimized simultaneously.
Overcoming Hardware and Pipeline Bottlenecks
While adding layers to the speculation hierarchy improves token acceptance rates, it introduces a new challenge: if the models execute in a strict "lockstep" fashion, the hardware sits idle while waiting for each stage to finish.
To realize the theoretical speedups of hierarchical decoding, recent architectures optimize how these models interact with hardware memory and execution pipelines:
- Asynchronous Pipelining: Rather than forcing models to wait for one another, frameworks like the one proposed by McDanel et al. (2025) break stage dependencies by transforming the hierarchy into an asynchronous pipeline. Multiple models of varying sizes operate in parallel across different segments of the token stream, maximizing GPU utilization.
- Sparse KV Caching: For long-sequence generation, memory bandwidth is choked not just by model weights, but by the Key-Value (KV) cache. The authors of TriForce (Sun et al. (2024)) resolved this by using a dual-mask hierarchy: a tiny draft model feeds into a version of the target model utilizing a highly compressed, retrieval-based sparse KV cache. This intermediate layer filters the draft before the full, exact KV cache of the target model performs the final lossless validation.
By aligning the mathematical structure of multi-stage verification with the physical realities of GPU memory bandwidth, hierarchical speculative decoding successfully mitigates the autoregressive bottleneck, providing a scalable path toward real-time LLM and LRM inference.
Key Foundational Work
Delay Aware Intelligent Transient Stability Assessment System (2017) — contributed 1 concept elements to this direction.
Soccer Videos Highlight Prediction and Annotation in Real Time (2005) — contributed 1 concept elements to this direction.