Large Language Models (LLMs) are notoriously slow because they are autoregressive: to generate a sentence, they must predict one token at a time. Each token requires a full forward pass through billions of parameters, a process that is heavily bottlenecked by memory bandwidth rather than raw computation.
The standard solution to this is speculative decoding, where a small "draft" model quickly guesses several future tokens, and the large "target" model verifies them all in a single parallel step. However, as noted by researchers like Zhenyu He and Heming Xia, this traditional approach has a major flaw: it requires training and maintaining a separate, specialized draft model for every target architecture, which is computationally expensive and difficult to deploy.
A new wave of research is moving toward "plug-and-play" acceleration—methods that achieve the speed of speculative decoding without the burden of auxiliary models or additional training.
The Model as its Own Drafter
The most intuitive way to eliminate the auxiliary model is to make the target model "speculate" on itself. This is known as self-speculative decoding.
The core insight, explored by Heming Xia in SWIFT, is that LLMs often contain redundant layers. By dynamically skipping intermediate layers during a "drafting" phase, the model can generate candidate tokens much faster than a full pass. Similarly, Fangcheng Liu and colleagues introduced Kangaroo, which uses a "double early-exiting" strategy. It exits the network early at a shallow layer to create a draft and then uses a token-level confidence threshold to decide when to stop drafting and start verifying.
These methods, including LEAP by LeiQuan Zheng, treat the model's internal feature transformations as a guide. If the features aren't changing much between layers, the model can safely "leap" ahead, saving time without losing the mathematical guarantees of the original output.
Model-Free Speculation via Retrieval
If we don't want to use a neural network for drafting at all, we can look at the text itself. Many sequences in LLM outputs are predictable—common phrases, code structures, or snippets from the input prompt.
Zhenyu He proposed REST (Retrieval-Based Speculative Decoding), which replaces the draft model with a simple datastore lookup. By retrieving common token sequences from a large corpus, the system can propose candidates that are highly likely to be correct. Yichao Fu took this further with Lookahead Decoding, which uses a Trie-tree to cache and reuse n-grams from the current generation session.
These "model-free" approaches, such as STAND by Woomin Song, are particularly effective for reasoning and coding tasks where structural patterns (like if __name__ == "__main__":) are repeated frequently. They exploit the idle compute power of modern GPUs to verify these "retrieved" strings in parallel, turning a memory-bound problem into a compute-bound one.
Dynamic Trees and Parallel Convergence
Even with a good drafter, the efficiency of speculative decoding depends on the "speculation length"—how many tokens you guess at once. Guess too few, and you waste parallel capacity; guess too many, and you waste time on tokens that will be rejected.
Yuhui Li and the EAGLE-2 team addressed this by using dynamic draft trees. Instead of a linear sequence of guesses, the drafter proposes a tree of possible paths. The target model then verifies the entire tree in one go. EAGLE-2 uses the draft model's own confidence scores to prune this tree in real-time, ensuring the target model only spends compute on high-probability branches.
For non-text modalities, such as image and video generation, researchers like Yao Teng have introduced Speculative Jacobi Decoding. This method treats the generation process as a system of equations that can be solved in parallel. By allowing tokens to "converge" toward their final values simultaneously, it breaks the sequential bottleneck of autoregressive visual models.
Decoupling the Drafter from the Target
A final hurdle in this field is portability. Traditionally, a draft model must share the exact same vocabulary as the target model. Nadav Timor and colleagues developed algorithms like SLEM to enable speculative decoding between models with heterogeneous vocabularies (Accelerating LLM Inference... for Heterogeneous Vocabularies).
This decoupling allows developers to use any off-the-shelf small model to accelerate any large model. Similarly, Huajun Bai introduced PARD, a framework where a single small draft model can be adapted to accelerate an entire family of larger models (like Llama 8B, 70B, and 405B) without needing separate training for each.
By moving away from rigid, co-trained model pairs and toward flexible, training-free, and self-referential mechanisms, this research direction is making high-speed LLM inference accessible to a much broader range of hardware and applications.