Attention Sinks and Outlier Features: A 'Catch, Tag, and Release' Mechanism for Embeddings

Stephen Zhang, Mustafa Khan, Vardan Papyan

In the field of Large Language Models (LLMs), "attention sinks" have long been observed as a curious phenomenon. These are specific tokens—often the initial token in a prompt or common punctuation like periods—that attract a disproportionate amount of attention from other tokens. While recognized as essential for model stability, their functional role has remained largely elusive.

A recent study from researchers at the University of Toronto and the Vector Institute, titled "Attention Sinks: A ‘Catch, Tag, Release’ Mechanism for Embeddings," proposes a structured functional explanation: LLMs use these sinks to "tag" information, much like marine biologists track wildlife. This mechanism allows models to propagate high-level semantic information across a sequence.

The 'Catch, Tag, and Release' (CTR) Mechanism

The researchers identify a three-stage process they call Catch, Tag, and Release (CTR). This system describes how attention sinks act as functional hubs within the transformer architecture:

  1. Catch: A sink token (e.g., a period or a speaker marker) attracts attention from a group of preceding or subsequent tokens.
  2. Tag: During the attention operation, the sink imprints its "value vector" onto the attending tokens. Because the sink's activation is often an outlier in magnitude, this "tag" becomes a dominant feature in the resulting embedding.
  3. Release: These tagged tokens are then released back into the model's residual stream. Because they now share a common feature direction (the tag), the model can treat them as a coherent group in subsequent layers, facilitating the propagation of global properties like sentiment or truthfulness.
The Attention mechanism allows sinks to broadcast their features across the sequence, effectively 'tagging' attending tokens with a shared feature direction.

Methodology and Empirical Evidence

To validate the CTR hypothesis, the authors employed a combination of visualization, quantification, and theoretical modeling:

  • PCA Visualization: Using Principal Component Analysis (PCA), the team visualized token embeddings across different layers. They found that tokens attending to the same sink begin to cluster together in the embedding space as they move deeper into the model.
  • Variance Quantification: They developed a metric to determine how much of an attention head's output variance is explained by the sink's "tag." In many models, just one or two tags can explain 20% to 40% of the total variance, sometimes reaching as high as 70%.
  • Semantic Probing: The researchers decomposed activations into "tag" and "non-tag" components. Probes trained on the tag component alone often outperformed those trained on full activations, suggesting that tags act as a "denoised" signal for high-level concepts.
PCA visualizations demonstrate how tokens exhibit no clustering at the input but begin to cluster according to their attended sink after the attention layer, a grouping that persists into deeper layers.

Key Insights

Sinks as Semantic Hubs

Probing experiments on datasets like Cities (True/False statements) and SST-2 (sentiment) revealed that tags are not merely numerical artifacts. On the Cities dataset, probes trained exclusively on the tag component achieved up to 99.5% accuracy in identifying truth values. This indicates that the CTR mechanism is a primary way models organize and move semantic information.

The Evolution of Reasoning Models

Interestingly, models fine-tuned for reasoning (such as DeepSeek-R1-distilled variants) exhibit a significantly more pronounced CTR mechanism. These models utilize more sinks, and their tags explain a higher percentage of embedding variance. Furthermore, while base models primarily use punctuation as sinks, reasoning models often repurpose semantically significant tokens (e.g., "Doctor" or speaker markers) as functional sinks.

Resilience to Normalization

The study found that sinks remain prevalent even in models using Query-Key Normalization (QK-Norm), a technique intended to suppress outlier activations. This suggests that the CTR mechanism is a fundamental architectural strategy rather than a byproduct of numerical instability.

Limitations and Future Directions

While the CTR framework provides a compelling explanation for attention sinks, the authors note several areas for future research. The current study shows a strong correlation between CTR and reasoning performance, but establishing a direct causal link remains a challenge. Additionally, the theoretical proofs are currently limited to simplified 2-layer transformer models.

The researchers suggest that understanding this "tagging" system could open doors for activation engineering. By manually manipulating these tag directions, it may be possible to steer model behavior more precisely, potentially improving factual accuracy and safety in future LLMs.

Summary updated

BibTeX
@inproceedings{zhang2025attention,
  author = {Stephen Zhang and Mustafa Khan and Vardan Papyan},
  title = {{Attention Sinks and Outlier Features: A 'Catch, Tag, and Release' Mechanism for Embeddings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=r8UWp9JeJi},
}
Figure