Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated Attention

Seojin Kim, Yehjin Shin, Noseong ParkarXiv 2026 · arXiv

TL;DR

This paper reveals that "attention sinks"—tokens that hog attention mass—manifest as structural outliers (spectral spikes) in a model's representation space. By applying a simple sigmoid gate after the attention mechanism, models selectively "squash" these spikes in the forward pass and act as a "gradient valve" in the backward pass, leading to significantly more stable training and cleaner internal representations.

What Problem Does This Paper Address?

Transformers have a "sink" problem. In many large language models (LLMs), a tiny subset of tokens—often just the very first token in a sequence—absorbs a massive, disproportionate amount of attention. This isn't just a quirk of visualization; these "attention sinks" are linked to "massive activations" (outlier values that can break quantization) and training instability.

Recently, researchers found a surprisingly simple fix: adding a sigmoid gate after the Scaled Dot-Product Attention (SDPA) output. This "gated attention" (now used in models like Qwen3) effectively kills the sink behavior, but until now, we didn't really know why. Is it just adding sparsity? Is it just shrinking everything? This paper moves past phenomenological observations to provide a rigorous structural explanation.

Methods

The authors bridge the gap between attention maps and representation geometry by recasting attention sinks as spectral objects.

  1. Token-Space Covariance: Instead of looking at attention weights ($A$), they look at the covariance of the post-attention output ($C_Y$). They show that a high-mass attention column (a sink) induces a specific, low-rank direction in this covariance matrix.
  2. Random Matrix Theory (RMT): They use the Marchenko–Pastur (MP) law and the Baik–Ben Arous–Péché (BBP) transition to define a "noise threshold." If a sink's influence is strong enough, it creates a "spectral spike"—an eigenvalue that pops out of the noise bulk as a detectable outlier.
  3. Dual Contraction Analysis: They mathematically prove that the post-SDPA gate acts as a "mode-dependent contraction."
    • Forward Pass: It selectively attenuates the energy of these sink-induced spectral spikes.
  4. Backward Pass: It acts as a "gradient valve," locally reducing the flow of gradients through sink columns, preventing the "gradient sinks" that lead to massive activations.

Datasets

The researchers validated their theory using several models and datasets:

  • Models: Qwen-1B (pre-trained) and Pythia-410M/2.8B (trained from scratch for this study).
  • Training Data: 15 billion tokens from FineWeb-Edu for the Pythia variants.
  • Evaluation Data: WikiText-103 for spectral diagnostics and a suite of zero-shot commonsense reasoning tasks (LAMBADA, PIQA, HellaSwag, etc.) to ensure performance remained competitive.

Key Insights and Ideas

  • Sinks are Structural, Not Just Visual: Attention sinks aren't just "hot spots" on a map; they are dominant low-rank modes that distort the entire geometry of the model's internal representations.
  • The "Gradient Valve" Effect: One of the most interesting "aha" moments is the backward pass analysis. Gating doesn't just change what the model sees; it changes how it learns. By filtering the value-path gradients, the gate prevents the feedback loop where sinks get stronger and stronger during training.
  • The AS-GS-MA Pathway: The paper connects three previously separate phenomena into a single chain: Attention Sinks $\rightarrow$ Gradient Sinks $\rightarrow$ Massive Activations. Gating breaks this chain at the first link.
  • Selectivity over Uniformity: A key finding is that the gate is selective. A simple scalar gate (shrinking everything equally) doesn't fix the spectral spikes. The learned, feature-dependent gate is necessary to target the specific directions in space where sinks reside.

Key Results

  • Sink Reduction: In gated models, first-token attention mass dropped from a staggering 46.7% to just 4.8%.
  • Spectral Cleanup: Gating significantly reduced "outlier mass" (eigenvalues above the MP threshold) and increased the effective rank of the representations, meaning the model uses its high-dimensional space more efficiently rather than being dominated by a few directions.
  • Massive Activation Suppression: In Pythia-410M, the maximum coordinate magnitude ($M_\infty$)—a proxy for activation spikes—was reduced from 320.0 to 67.4 by adding the gate.
  • Training Stability: Gated models showed much lower "representation dominance" throughout the training process, preventing the emergence of the distorted geometry that often precedes training crashes.

Limitations and Future Work

While the paper provides a strong mechanistic account, the authors acknowledge a few areas for further exploration:

  • Isotropic Assumptions: The RMT analysis relies on an idealized "isotropic" (uniform) noise model. Real-world data is more complex, and while the theory holds up empirically, the mathematical mapping isn't perfectly 1:1.
  • Local vs. Global Dynamics: The gradient analysis is "local" (looking at a single step). A full theory of how gating affects the global trajectory of training over millions of steps remains an open challenge.
  • Non-Sink Outliers: The gate might also suppress useful, non-sink outliers (like rare but semantically crucial tokens). Future work could investigate if there's a "sweet spot" for gating that preserves beneficial signal while killing the sink noise.

Summary updated

Abstract

Transformer attention exhibits attention sinks, where a small subset of source tokens absorbs disproportionate attention mass across layers. Recent work shows that post-SDPA output gating strongly reduces attention-sink behavior, while separate studies link attention sinks to gradient sinks and massive activations through backward training dynamics. Yet how the same gate affects forward sink structure, backward gradient concentration, and training-time representation signatures remains unclear. We address this gap by formulating attention sinks as sink-induced low-rank modes in token-space covariance, with directions aligned to high-mass attention columns. Under this view, post-SDPA gating acts as a mode-dependent contraction: it attenuates sink-induced spectral energy in the forward pass and locally reduces value-path gradients routed through sink columns in the backward pass. Empirically, random-matrix diagnostics show that gating reduces outlier mass, weakens sink-subspace alignment, and increases effective rank, while backward analyses show active-gate attenuation of first-token value gradients. Training-time diagnostics further show that gating suppresses the co-emergence of attention concentration, value-gradient amplification, and representation dominance.
BibTeX
@misc{kim2026attention,
  author = {Seojin Kim and Yehjin Shin and Noseong Park},
  title = {{Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated Attention}},
  year = {2026},
  eprint = {2611.20403},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20403},
}
Figure