Attention Sinks and Structural Anchoring in Transformers

63 concepts46 papers2 proposals

When analyzing the internal attention maps of Large Language Models (LLMs), an unexpected pattern consistently emerges: models allocate a massive, seemingly disproportionate amount of attention weight to the very first tokens of a sequence. This happens even when those initial tokens are completely devoid of semantic meaning, such as a standalone beginning-of-sentence ([BOS]) marker or a punctuation mark.

This phenomenon is not a training flaw, but a mathematical necessity of the Transformer architecture. Over recent years, researchers have discovered that these "attention sinks" serve as vital structural anchors. Understanding why they form and how they stabilize internal representations has unlocked new techniques for infinite-context window streaming, aggressive memory compression, and robust long-context training.

The Softmax Constraint and the Birth of Attention Sinks

To understand why attention sinks form, we must look at the mathematics of the self-attention mechanism. For every query token in a layer, the model computes attention scores across all previous key tokens and applies the Softmax function. Crucially, Softmax forces the final attention weights to sum exactly to one.

What happens when a query layer processes a token that does not strongly relate to any previous token in the context? The model still must allocate its attention weight somewhere to satisfy the Softmax constraint. As demonstrated in the foundational work on StreamingLLM, the model naturally learns to dump this excess, unallocated attention weight onto the very first tokens of the sequence.

The stability provided by these initial tokens is primarily structural rather than semantic. Experiments in StreamingLLM show that replacing the starting text with neutral characters or line breaks does not degrade performance; the model simply relies on their fixed initial position to anchor the attention distribution.

From a geometric perspective, Cancedda reveals that these initial tokens host high-norm signals in non-vocabulary "dark" subspaces, acting as universal sinks that prevent irrelevant context from corrupting subsequent representations. Similarly, Wertheimer et al. describe this anchoring as designating a specific token as a low-norm reference point near the latent space origin. This provides a default destination for queries that lack relevant keys, preventing information from over-mixing and degrading across long sequences.

The Catastrophe of Eviction and Feature Drift

The realization that initial tokens act as structural anchors explains a long-standing failure mode in long-context deployment. To handle long or streaming text inputs under hardware memory constraints, a natural instinct is to implement a sliding window over the Key-Value (KV) cache, evicting older tokens as new ones are generated.

However, as documented by Wang et al. and the authors of StreamingLLM, the moment the very first tokens of a sequence are evicted from the cache, the model experiences a catastrophic performance collapse, and perplexity explodes.

Without the initial attention sink, queries that lack strong semantic matches are forced to distribute their weights across the remaining local tokens within the sliding window. Han explains this as a failure of the internal feature distribution: discarding the initial anchor tokens causes the hidden states to drift into out-of-distribution regions, leading to immediate model failure.

Furthermore, the integrity of the entire autoregressive generation process depends heavily on these initial representations. As Jaiswal et al. observe, processing the beginning of a sequence with full model capacity anchors the attention mechanism, acting as a warm-up period that allows for more aggressive computational skipping or optimization in later stages without sacrificing coherence.

Anchor-Preserving KV Cache Compression

Recognizing that attention sinks act as structural anchors allows engineers to compress the memory footprint of LLMs during inference without requiring expensive retraining.

The core recipe, surveyed by Li et al., is straightforward: maintain a small, fixed set of initial attention sink tokens permanently in the KV cache, alongside a rolling window of recent context tokens. This simple modification allows models to process virtually infinite text streams with stable perplexity.

Advanced compression frameworks build directly on this intuition to optimize memory:

  • Adaptive Eviction: FastGen, introduced by Ge et al., profiles attention heads to dynamically identify and prioritize structural anchors (like start-of-sentence markers and punctuation), allowing for aggressive eviction of intermediate, non-essential cache data.
  • Static Instruction Preservation: Wu et al. utilize static anchor preservation to ensure that high-importance tokens and system instructions are never evicted during long-sequence generation, keeping the model aligned with its original prompt.
  • High-Precision Quantization: Because attention sinks capture a disproportionate amount of attention weight, they are highly sensitive to quantization errors. Yuan et al. developed KVSink, which detects these sinks as "stable outliers" in early layers and isolates them for high-precision storage, while compressing the rest of the memory-heavy KV cache to lower bit-widths.

Designing and Repurposing Structural Anchors

Beyond merely preserving naturally occurring sinks, recent work focuses on actively designing or repurposing structural anchors to control and stabilize attention flow.

Rather than relying on the first word of a user's text to accidentally become the attention sink, the authors of StreamingLLM propose intentionally introducing a dedicated, non-semantic placeholder token during the pre-training phase. This gives the model a clean, dedicated destination to dump excess attention scores from day one.

Conversely, SinkTrack, developed by Liu et al., repurposes the naturally occurring attention sink. Because the model already allocates massive attention weights to the first token, the authors inject external context or critical instructions directly into this sink. This anchors the source material throughout long-form generation, preventing the model's focus from drifting toward recently generated text and significantly reducing hallucinations.

Standard Sliding Window (Catastrophic Collapse):
[Evicted BOS] -> [Token 10] -> [Token 11] -> [Token 12] (No sink; attention over-mixes)

Anchor-Preserving Window (Stable Processing):
[BOS (Sink)]  -> [Token 10] -> [Token 11] -> [Token 12] (Excess attention dumps safely into BOS)

Anchors can also be distributed throughout a sequence to maintain parity across long contexts. Begin et al. introduce Pause-Tuning, a method that inserts discrete <PAUSE> tokens into long sequences. These markers act as localized attention anchors, creating spikes in attention distribution that interrupt the typical decay of focus over extended contexts and refresh the model's processing state.

Finally, anchoring plays a crucial role in maintaining numerical stability within position embeddings. Du et al. discovered that Rotary Positional Embeddings (RoPE) break down numerically when paired with BFloat16 precision during long-context training. Their solution, AnchorAttention, treats the first token as a shared global reference point with a fixed position index of zero. This anchor remains visible to every other token in the batch, providing a stable numerical baseline that prevents geometric breakdown across massive sequence lengths.

Key Foundational Work

FCM: Forgetful Causal Masking Makes Causal Language Models Better Zero-Shot Learners (2022) — contributed 1 concept elements to this direction.
Outlier Dimensions that Disrupt Transformers are Driven by Frequency (2022) — contributed 1 concept elements to this direction.
Exploring the Role of BERT Token Representations to Explain Sentence Probing Results (2021) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 49 papers
  2. Version 1 · 2026-10-04 · 75 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_attention_sink_preservation_54f45733,
  title = {attention sink preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-sink-preservation/rdr_54f457334891474bb92e16fd24b4303a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_attention_sink_preservation_54f45733_v2,
  title = {attention sink preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-sink-preservation/rdr_54f457334891474bb92e16fd24b4303a/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake