As Transformers scale to handle hundreds of thousands of tokens, the attention mechanism—the core engine of their success—faces a mathematical crisis. In standard attention, as the sequence length $n$ grows, the softmax distribution naturally tends to flatten, spreading its focus across too many tokens. This "attention dilution" makes it increasingly difficult for a model to retrieve specific information, a problem often described as finding a needle in an ever-expanding haystack.
To combat this, a growing body of research suggests that the key to stable long-context modeling lies in logarithmic scaling. By adjusting attention scores or temperatures relative to $\log n$, researchers are finding they can prevent representational collapse and maintain sharp focus even as context windows reach extreme lengths.
The Problem of Attention Dilution
In a standard Transformer, the attention score for a token is calculated via a softmax over dot-products. As Xuechen Zhang and colleagues observe in Selective Attention: Enhancing Transformer through Principled Context Control, this "one-size-fits-all" approach leads to a "wash-out" effect. When the number of distractors increases, the probability mass assigned to the relevant "target" token vanishes.
Rachit Bansal and others characterize this in Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs as a requirement for the "logit margin." To keep the signal-to-noise ratio stable, the difference between the score of a relevant token and a distractor must grow at a rate of $O(\ln n)$. Without this adjustment, the model's focus becomes overly dispersed, leading to what Muhammad Ahmed Mohsin calls "retrieval fragility" in On the Fundamental Limits of LLMs at Scale.
The Critical Threshold of Logarithmic Scaling
Why $\log n$? The intuition comes from the mathematical stability of the softmax function. Yury Polyanskiy, Philippe Rigollet, and their team prove in Critical attention scaling in long-context transformers that $\beta_n \sim \log n$ is a "critical" threshold.
If you scale the attention temperature more slowly than $\log n$, the model undergoes a phase transition into rank collapse, where all token representations clump together into a single, uninformative cluster. If you scale it faster, the model becomes too rigid, acting like a simple identity map. Logarithmic scaling sits at the "Goldilocks" zone, ensuring the attention remains content-adaptive and expressive.
This is echoed by Philippe Rigollet in The Mean-Field Dynamics of Transformers, which treats tokens as physical particles moving on a sphere. To keep these particles from collapsing into a single point as more are added to the system, the "interaction energy" (attention) must be tempered by the logarithm of the particle count.
Stabilizing Entropy and Sparsity
A practical way to view this scaling is through the lens of attention entropy—a measure of how "spread out" the model's focus is. Junlong Li and Pengfei Liu demonstrate in Extending LLMs' Context Window with 100 Samples that by applying dynamic logit scaling, they can stabilize this entropy. This allows models like LLaMA-2 to expand their context window significantly with minimal additional training.
Similarly, Marcos Vinicius Treviso and Pavlo Vasylenko propose ASEntmax in Long-Context Generalization with Sparse Attention. They use a learnable scaling factor based on $\log n$ to maintain a consistent level of sparsity. By assigning exact zeros to irrelevant tokens, they ensure that the "probability mass" doesn't leak away into the background noise of a long sequence.
Implementation Across Architectures
The need for logarithmic scaling isn't limited to standard softmax attention. It appears as a fundamental requirement across various architectural tweaks:
- Sigmoid Attention: Jason Ramapuram and Federico Danieli show in Theory, Analysis, and Best Practices for Sigmoid Self-Attention that replacing softmax with sigmoid requires a bias term of $-\log n$. This bias emulates the normalization properties of softmax, preventing "exploding attention norms" as sequences grow.
- Vision Transformers: In Win-Win: Training High-Resolution Vision Transformers from Two Windows, researchers use temperature adjustments to compensate for the jump from low-resolution training windows to high-resolution full images, preventing the model from becoming overly concentrated on a few patches.
- Position Embeddings: Arthur S. Bianchessi and colleagues use a "Scalable Softmax" in their Bayesian Attention Mechanism to enable 500x context extrapolation, treating the sequence length as a prior that must be accounted for during the attention calculation.
Conclusion
The transition from fixed-length training to "infinite" context inference requires more than just better hardware; it requires a fundamental shift in how we scale the interactions between tokens. By anchoring attention scores to the logarithm of the sequence length, researchers are providing Transformers with the "concentration resilience" needed to navigate massive volumes of data without losing their way. As Alireza Naderi suggests in Mind the Gap, maintaining the "spectral gap" of the attention matrix through proper scaling is what ultimately prevents the model from collapsing into a state where all tokens look the same.