Attention Sinks Induce Gradient Sinks

Yihong Chen, Quanming YaoarXiv 2026 · DOI · arXiv

TL;DR

This paper reveals that the "massive activations" (numerical outliers) often seen in Large Language Models are actually an adaptive defense mechanism. By studying backpropagation, the authors show that "attention sinks" create intense "gradient sinks" on the first token; the model then grows massive activations to dampen this localized gradient pressure and stabilize training.

What Problem Does This Paper Address?

If you look under the hood of a modern Transformer like Llama, you’ll find two strange phenomena, usually occurring on the very first token of a sequence:

  1. Attention Sinks (AS): A few tokens (especially the first one) attract a massive, disproportionate amount of attention, even if they aren't semantically relevant.
  2. Massive Activations (MA): The internal numerical values (activations) for these same tokens become enormous—far larger than any other tokens in the sequence.

Researchers have known these two are correlated, but the "why" remained fuzzy. Most theories focused on the forward pass (how the model computes an output). This paper asks a deeper question about the training process: Why does the optimization process "choose" to create these massive numerical outliers in the first place, especially when modern architectures use normalization (like RMSNorm) that should theoretically make raw magnitudes irrelevant?

Methods

The authors shift the focus from the forward pass to backpropagation. Their approach combines three elements:

  • Empirical Tracking: They trained Llama-like models (0.1B and 0.3B parameters) from scratch and tracked the "gradient pressure" on every token at every step of training.
  • Theoretical Analysis: They mathematically proved that under a causal mask (where tokens only look at previous ones), the first token naturally becomes a "Gradient Sink" (GS), accumulating gradients from every subsequent token in the sequence.
  • V-scale Intervention: To prove that MA is a response to GS, they introduced a novel modification called V-scale. This acts as a "gradient valve" on the value-path in the attention layer. It suppresses small-norm values during the forward pass, which mathematically attenuates the backpropagated gradients for those tokens without significantly changing the model's logic.

Datasets

The models were pretrained from scratch on the C4 (Colossal Clean Crawled Corpus) dataset. The authors used standard Llama-style configurations (RoPE, SwiGLU, RMSNorm) to ensure their findings apply to the architectures most commonly used in the industry today.

Key Insights and Ideas

  • The "Gradient Sink" Discovery: Because of the causal mask, the first token is the only one visible to every subsequent token. During backprop, the gradients from all future tokens aggregate at the first token. This creates a "pressure cooker" environment for the first token's weights.
  • MA as an Adaptive Response: In architectures using RMSNorm, the gradient sent back through a layer is roughly inversely proportional to the activation's norm. The authors argue that the model "learns" to increase the activation magnitude (MA) of the first token specifically to survive this gradient pressure. By making the activation massive, the model effectively "muffles" the incoming gradient signal, preventing it from destabilizing the weights.
  • The Asymmetry of QKV: The gradient concentration is most severe on the Key (K) and Value (V) pathways, while the Query (Q) pathway remains flat. This explains why outliers are token-specific rather than position-agnostic.
  • Decoupling AS and MA: Previously, many thought MA was required for AS to work. This paper proves they can be separated. With the V-scale intervention, the model maintains its attention sinks (the "no-op" behavior) but no longer needs to grow massive activations because the gradient pressure has been relieved by the "valve."

Key Results

  • Gradient Concentration: In baseline models, the gradient norm on the first token's value path is several times larger than on any other token.
  • V-scale Success: Models trained with V-scale showed a significant reduction in activation outliers. For example, in the 0.3B model, the MLP output norms for the first token were drastically lower compared to the baseline, while the "Sink Rate" (how much attention the first token attracts) remained high or even increased.
  • Stability: V-scale models achieved slightly lower validation losses than baselines, suggesting that managing these gradient sinks might actually lead to better-trained models.

Limitations and Future Work

  • Model Scope: The study focused on dense, Llama-like models. It remains to be seen if the same "Gradient Sink" mechanics apply to Mixture-of-Experts (MoE) architectures or multi-modal models.
  • Scale: The experiments were conducted on relatively small models (up to 0.3B parameters). While the mechanics are likely the same, the extreme numerical instabilities of 70B+ parameter models might introduce additional nuances.
  • Downstream Impact: The paper is a mechanistic study. The authors did not perform an exhaustive battery of downstream task evaluations (like MMLU) to see if suppressing massive activations improves or harms specific reasoning capabilities.

Summary updated

Abstract

Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing studies have largely focused on the forward pass, making it unclear whether their connection is direct or mediated by a training-time mechanism. We study this question from the perspective of backpropagation. Empirically and theoretically, we show that under causal mask, attention sinks can induce pronounced gradient concentration, which we term gradient sinks. Furthermore, in pre-norm architectures with RMSNorm, massive activations can be understood as an adaptive response to this localized gradient pressure during training. To test this hypothesis, we introduce V-scale, a modification that adjusts value-path backpropagated gradients. In pretrained V-scale models, attention sinks are preserved whereas massive activations are suppressed. These results support the interpretation that gradient sink is a key training-time mediator linking attention sinks and massive activations.
BibTeX
@misc{chen2026attention,
  author = {Yihong Chen and Quanming Yao},
  title = {{Attention Sinks Induce Gradient Sinks}},
  year = {2026},
  eprint = {2603.17771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.17771},
}
Figure