In the quest to scale Large Language Models (LLMs) to handle millions of tokens, researchers have encountered a puzzling phenomenon: if you remove the very first token of a sequence—even if it is just a semantically useless "beginning-of-sentence" (BOS) marker—the model’s performance often collapses instantly. This discovery has led to the concept of Attention Sinks, a structural mechanism that LLMs use to maintain numerical stability.
The Softmax Bottleneck
To understand why attention sinks exist, we must look at the Softmax function at the heart of the Transformer. In every attention layer, the scores for all previous tokens must sum to exactly one. As Beidi Chen explains in StreamingLLM, even when a query finds no relevant information in the preceding context, the Softmax requirement forces the model to allocate its attention probability somewhere.
Because the first token is visible to every subsequent token in a causal sequence, the model learns to use it as a "landing spot" or "trash can" for this excess attention mass. This is not because the first token is semantically important, but because it is a persistent, reliable numerical anchor. Ivan Kobyzev and colleagues in the Integral Transformer describe this as a "Goldilocks" balance: the model needs these sinks to filter out noise, but over-reliance on them can drown out the actual signal.
Why Models Collapse Without Sinks
The importance of these anchors becomes visible when implementing Sliding Window Attention. In a naive sliding window, as the model generates new text, the oldest tokens are evicted from the Key-Value (KV) cache to save memory. However, Beidi Chen demonstrated that as soon as the very first token is evicted, the perplexity of the model spikes to infinity.
Without the initial "sink" token to absorb the mandatory Softmax probability, the attention scores are forced onto other, potentially irrelevant tokens in the current window. This destabilizes the entire numerical distribution of the layer. Mechanistic interpretability work on the Repeated Token Phenomenon has even identified specific neural circuits—a "detector" layer and an "amplifier" layer—dedicated solely to identifying the initial token and boosting its representation to serve as this sink.
Adaptive Compression and Anchoring
Recognizing that not all tokens are created equal has allowed researchers to build much more efficient inference systems. Instead of treating the KV cache as a uniform buffer, frameworks like FastGen, developed by Suyu Ge and Yunan Zhang, profile attention heads to identify which ones rely on these anchors. By preserving "attention anchors" (like the BOS token) while aggressively pruning others, they reduce memory usage by 50% without retraining.
Similarly, Liming Lu and the team behind DynaKV found that learned gating mechanisms automatically assign the highest retention rates to these initial tokens. This confirms that the model's internal logic is fundamentally tethered to these starting points.
Beyond Text: Multimodal and Positional Stability
The sink phenomenon extends beyond standard language modeling:
- Multimodal Stability: In SEED-Story, researchers found that image boundary markers act as multimodal attention sinks. Retaining them is critical to prevent the sudden degradation of image quality in long-form generation.
- Positional Encoding: Jiaheng Liu showed in E2-LLM that keeping the position offsets of the first few tokens at zero—even when the rest of the sequence is shifted—maintains the stability of the attention mechanism during extreme context extension.
- Geometric Separation: Davis Wertheimer argues in Frayed RoPE that attention sinks serve a geometric purpose, acting as a low-norm reference point near the latent space origin to prevent information "over-mixing" in long sequences.
Sinks as Diagnostic Tools
Because attention sinks are so central to model stability, they can also be used to monitor model health. Jakub Binkowski introduced SinkProbe, which detects hallucinations by monitoring the flow of attention into these sinks. A collapse in the expected "sink" pattern often signals that the model's internal information flow has broken down, leading to factual errors.
By treating the first tokens of a sequence as permanent structural anchors rather than just the start of a sentence, the field has moved toward "infinite" context windows and significantly more efficient hardware utilization. The "sink" is not a bug of the Transformer architecture, but a feature that provides the numerical foundation for stable reasoning.