Attention Drift: What Autoregressive Speculative Decoding Models Learn

Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek, Kaan Oktay, Hongyang Zhang, Stephen XiaarXiv 2026 · arXiv

TL;DR

Speculative decoding speeds up LLM inference using a smaller "drafter" model, but these drafters are notoriously fragile when faced with altered prompt templates or long contexts. This paper identifies attention drift—a phenomenon where the drafter's attention shifts away from the prompt and onto its own recent outputs due to unnormalized hidden state growth. By introducing a simple post-normalization architecture, the authors stabilize the drafter, making it significantly more robust and computationally efficient to train.

What Problem Does This Paper Address?

Speculative decoding is a popular technique to accelerate LLM inference. It uses a lightweight "drafter" model to quickly guess a sequence of future tokens, which a larger "verifier" model then validates in a single parallel pass.

However, existing drafters are incredibly sensitive to their deployment environment. If you slightly tweak the chat template, remove a beginning-of-sequence (BoS) token, or feed the model a long context, the drafter's acceptance rate plummets. To combat this, engineers frequently have to retrain specialized drafters for specific prompts or engines. This paper investigates the root cause of this fragility, uncovering a structural flaw in how modern drafters learn.

Methods

The authors investigate popular autoregressive drafter designs, specifically EAGLE-3 and Multi-Token Prediction (MTP) heads. By analyzing the hidden-state dynamics across successive speculation steps, they discover that the hidden-state magnitudes grow monotonically with chain depth.

To fix this, they propose two simple architectural modifications to the standard pre-norm drafter design:

  1. Post-normalization (Post-norm): Moving the RMSNorm layer to the output of the drafter's transformer layer before the residual connection. This prevents the scale of the representations from compounding across speculation steps.
  2. Per-hidden-state RMSNorm: Applying individual normalization layers to the target model's low, mid, and high hidden states before fusing them into the drafter input, preventing high-layer signals from dominating the input.
Figure 2: Overview of Pre-norm (Left) and proposed Post-norm (Right) architecture.

Datasets

The models were trained and evaluated using the following datasets:

  • Training: The Open-PerfectBlend dataset (1.4M samples) was used for Llama and Qwen3 variants, and the Nemotron post-training dataset (1.4M samples) was used for Qwen3.5 and GPT-oss variants.
  • Evaluation: The authors used a wide suite of standard benchmarks: MT-Bench (multi-turn chat), GSM8K and MATH-500 (mathematics), HumanEval and LiveCodeBench (coding), Alpaca, GPQA, and LongBench (for long-context evaluation).

Key Insights and Ideas

  • Layer-Stacking vs. Autoregression: Standard pre-norm drafters don't actually behave like independent autoregressive models. Because their unnormalized residual paths allow hidden states to explode in magnitude, the drafter implicitly learns a depth-dependent refinement process. It essentially acts as if you are stacking extra transformer layers on top of the target model, rather than generating text autoregressively.
  • The Mechanics of Attention Drift: As the speculation chain deepens, the exploding hidden state magnitudes cause the drafter's attention to progressively drift away from the prompt's "sink token" (the stable anchor at the start of a sequence) and concentrate heavily on its own recently drafted tokens.
Figure 1: Attention drift. During speculation, the drafter’s attention moves from the prompt’s sink token onto its own recently-generated tokens. Left: Emergence of the attention sink demonstrated on drafter’s attention heatmap, rows = query, columns = key; darker = higher attention. Center: Graphical visualization of attention drift on a drafter. Right: Attention per token position on x axis, with speculated tokens to the right of the dashed line.
  • Drift is a Symptom, Not the Cause: Interestingly, when the authors tried using "gated attention" to artificially eliminate the attention sink and stop the drift, the hidden state magnitudes still exploded, and performance remained fragile. This proved that attention drift is merely a symptom of the underlying unnormalized residual dynamics.
  • Zero-Shot Depth Generalization: By enforcing a stable, depth-invariant hidden-state distribution via Post-norm, the drafter is forced to learn a true autoregressive prediction function. As a result, a drafter trained on a short speculation sequence (e.g., depth 2) can generalize perfectly to much longer sequences at inference time.

Key Results

  • Template Resilience: Under severe template perturbations (such as removing the chat template and BoS tokens entirely), the Post-norm architecture improved token acceptance length by up to 2× over the current state-of-the-art pre-norm EAGLE3.
  • Long-Context Performance: On LongBench tasks, Post-norm outperformed Pre-norm by 20–25% (1.18× improvement) across all categories.
  • Standard Benchmarks: Post-norm achieved a consistent 1.10× average improvement across seven standard benchmarks spanning chat, math, and coding.
  • Training Speedup: Because Post-norm generalizes so well across sequence depths, the authors successfully reduced the train-time-test (TTT) depth from 8 to 4. This cut total training time by roughly one-third without any degradation in final performance.

Limitations and Future Work

  • Architecture Scope: The paper focuses its primary diagnosis and architectural interventions on EAGLE-3. While the authors observed attention drift in MTP heads, they did not deeply evaluate their proposed fixes across MTP or other architectures like Hydra.
  • Scale Constraints: Due to compute limitations, all evaluations were restricted to models under 120B parameters; the behavior of attention drift at a massive scale (e.g., 400B+ models) remains unverified.
  • Proprietary Training Data: The exact training procedure for commercial MTP heads (like Qwen3.5) is undisclosed, leaving the precise interplay between training loss and architecture for future study.

Summary updated

Abstract

Speculative decoding accelerates LLM inference by drafting future tokens with a small model, but drafter models degrade sharply under template perturbation and long-context inputs. We identify a previously-unreported phenomenon we call \textbf{attention drift}: as the drafter generates successive tokens within a speculation chain, attention progressively moves from the prompt onto its own recently-generated tokens. We observe this across both \emph{EAGLE3} drafters and \emph{MTP heads}, suggesting drift is a property of drafter designs. We trace this to the un-normalized residual path between chain steps: the drafter's hidden state magnitude grows monotonically with chain depth, which exhibits dynamics consistent with additional pre-norm transformer layers stacked on the target rather than as a standalone autoregressive predictor. In order to limit the growth, we propose two architectural changes: Post-norm on the drafter hidden states and per-hidden-state RMSNorm after capturing target hidden states. Our interventions improve acceptance length over the current leading model, pre-norm EAGLE3, by up to $2\times$ under template perturbation, $1.18\times$ on long-context tasks, and $1.10\times$ on seven standard benchmarks spanning multi-turn chat, math, and coding. Our changes also allow shorter train-time-test depths to generalize over longer drafting sequences.
BibTeX
@misc{eldenk2026attention,
  author = {Doğaç Eldenk and Payal Mohapatra and Yigitcan Comlek and Kaan Oktay and Hongyang Zhang and Stephen Xia},
  title = {{Attention Drift: What Autoregressive Speculative Decoding Models Learn}},
  year = {2026},
  eprint = {2605.09992},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.09992},
}
Figure