TL;DR
Speculative decoding speeds up LLM inference using a smaller "drafter" model, but these drafters are notoriously fragile when faced with altered prompt templates or long contexts. This paper identifies attention drift—a phenomenon where the drafter's attention shifts away from the prompt and onto its own recent outputs due to unnormalized hidden state growth. By introducing a simple post-normalization architecture, the authors stabilize the drafter, making it significantly more robust and computationally efficient to train.
What Problem Does This Paper Address?
Speculative decoding is a popular technique to accelerate LLM inference. It uses a lightweight "drafter" model to quickly guess a sequence of future tokens, which a larger "verifier" model then validates in a single parallel pass.
However, existing drafters are incredibly sensitive to their deployment environment. If you slightly tweak the chat template, remove a beginning-of-sequence (BoS) token, or feed the model a long context, the drafter's acceptance rate plummets. To combat this, engineers frequently have to retrain specialized drafters for specific prompts or engines. This paper investigates the root cause of this fragility, uncovering a structural flaw in how modern drafters learn.
Methods
The authors investigate popular autoregressive drafter designs, specifically EAGLE-3 and Multi-Token Prediction (MTP) heads. By analyzing the hidden-state dynamics across successive speculation steps, they discover that the hidden-state magnitudes grow monotonically with chain depth.
To fix this, they propose two simple architectural modifications to the standard pre-norm drafter design:
- Post-normalization (Post-norm): Moving the RMSNorm layer to the output of the drafter's transformer layer before the residual connection. This prevents the scale of the representations from compounding across speculation steps.
- Per-hidden-state RMSNorm: Applying individual normalization layers to the target model's low, mid, and high hidden states before fusing them into the drafter input, preventing high-layer signals from dominating the input.
Datasets
The models were trained and evaluated using the following datasets:
- Training: The Open-PerfectBlend dataset (1.4M samples) was used for Llama and Qwen3 variants, and the Nemotron post-training dataset (1.4M samples) was used for Qwen3.5 and GPT-oss variants.
- Evaluation: The authors used a wide suite of standard benchmarks: MT-Bench (multi-turn chat), GSM8K and MATH-500 (mathematics), HumanEval and LiveCodeBench (coding), Alpaca, GPQA, and LongBench (for long-context evaluation).
Key Insights and Ideas
- Layer-Stacking vs. Autoregression: Standard pre-norm drafters don't actually behave like independent autoregressive models. Because their unnormalized residual paths allow hidden states to explode in magnitude, the drafter implicitly learns a depth-dependent refinement process. It essentially acts as if you are stacking extra transformer layers on top of the target model, rather than generating text autoregressively.
- The Mechanics of Attention Drift: As the speculation chain deepens, the exploding hidden state magnitudes cause the drafter's attention to progressively drift away from the prompt's "sink token" (the stable anchor at the start of a sequence) and concentrate heavily on its own recently drafted tokens.
- Drift is a Symptom, Not the Cause: Interestingly, when the authors tried using "gated attention" to artificially eliminate the attention sink and stop the drift, the hidden state magnitudes still exploded, and performance remained fragile. This proved that attention drift is merely a symptom of the underlying unnormalized residual dynamics.
- Zero-Shot Depth Generalization: By enforcing a stable, depth-invariant hidden-state distribution via Post-norm, the drafter is forced to learn a true autoregressive prediction function. As a result, a drafter trained on a short speculation sequence (e.g., depth 2) can generalize perfectly to much longer sequences at inference time.
Key Results
- Template Resilience: Under severe template perturbations (such as removing the chat template and BoS tokens entirely), the Post-norm architecture improved token acceptance length by up to 2× over the current state-of-the-art pre-norm EAGLE3.
- Long-Context Performance: On LongBench tasks, Post-norm outperformed Pre-norm by 20–25% (1.18× improvement) across all categories.
- Standard Benchmarks: Post-norm achieved a consistent 1.10× average improvement across seven standard benchmarks spanning chat, math, and coding.
- Training Speedup: Because Post-norm generalizes so well across sequence depths, the authors successfully reduced the train-time-test (TTT) depth from 8 to 4. This cut total training time by roughly one-third without any degradation in final performance.
Limitations and Future Work
- Architecture Scope: The paper focuses its primary diagnosis and architectural interventions on EAGLE-3. While the authors observed attention drift in MTP heads, they did not deeply evaluate their proposed fixes across MTP or other architectures like Hydra.
- Scale Constraints: Due to compute limitations, all evaluations were restricted to models under 120B parameters; the behavior of attention drift at a massive scale (e.g., 400B+ models) remains unverified.
- Proprietary Training Data: The exact training procedure for commercial MTP heads (like Qwen3.5) is undisclosed, leaving the precise interplay between training loss and architecture for future study.
Summary updated