Logarithmic Scaling for Stable Long Context Attention

This direction is split into other directions: logarithmic attention scaling, logarithmic temperature scaling. Its last version stays readable and citable here.

Version 1 of 1216 papers362 conceptsfrozen 2026-10-04

As Transformers scale to handle hundreds of thousands of tokens, the attention mechanism—the core engine of their success—faces a mathematical crisis. In standard attention, as the sequence length $n$ grows, the softmax distribution naturally tends to flatten, spreading its focus across too many tokens. This "attention dilution" makes it increasingly difficult for a model to retrieve specific information, a problem often described as finding a needle in an ever-expanding haystack.

To combat this, a growing body of research suggests that the key to stable long-context modeling lies in logarithmic scaling. By adjusting attention scores or temperatures relative to $\log n$, researchers are finding they can prevent representational collapse and maintain sharp focus even as context windows reach extreme lengths.

The Problem of Attention Dilution

In a standard Transformer, the attention score for a token is calculated via a softmax over dot-products. As Xuechen Zhang and colleagues observe in Selective Attention: Enhancing Transformer through Principled Context Control, this "one-size-fits-all" approach leads to a "wash-out" effect. When the number of distractors increases, the probability mass assigned to the relevant "target" token vanishes.

Rachit Bansal and others characterize this in Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs as a requirement for the "logit margin." To keep the signal-to-noise ratio stable, the difference between the score of a relevant token and a distractor must grow at a rate of $O(\ln n)$. Without this adjustment, the model's focus becomes overly dispersed, leading to what Muhammad Ahmed Mohsin calls "retrieval fragility" in On the Fundamental Limits of LLMs at Scale.

The Critical Threshold of Logarithmic Scaling

Why $\log n$? The intuition comes from the mathematical stability of the softmax function. Yury Polyanskiy, Philippe Rigollet, and their team prove in Critical attention scaling in long-context transformers that $\beta_n \sim \log n$ is a "critical" threshold.

If you scale the attention temperature more slowly than $\log n$, the model undergoes a phase transition into rank collapse, where all token representations clump together into a single, uninformative cluster. If you scale it faster, the model becomes too rigid, acting like a simple identity map. Logarithmic scaling sits at the "Goldilocks" zone, ensuring the attention remains content-adaptive and expressive.

This is echoed by Philippe Rigollet in The Mean-Field Dynamics of Transformers, which treats tokens as physical particles moving on a sphere. To keep these particles from collapsing into a single point as more are added to the system, the "interaction energy" (attention) must be tempered by the logarithm of the particle count.

Stabilizing Entropy and Sparsity

A practical way to view this scaling is through the lens of attention entropy—a measure of how "spread out" the model's focus is. Junlong Li and Pengfei Liu demonstrate in Extending LLMs' Context Window with 100 Samples that by applying dynamic logit scaling, they can stabilize this entropy. This allows models like LLaMA-2 to expand their context window significantly with minimal additional training.

Similarly, Marcos Vinicius Treviso and Pavlo Vasylenko propose ASEntmax in Long-Context Generalization with Sparse Attention. They use a learnable scaling factor based on $\log n$ to maintain a consistent level of sparsity. By assigning exact zeros to irrelevant tokens, they ensure that the "probability mass" doesn't leak away into the background noise of a long sequence.

Implementation Across Architectures

The need for logarithmic scaling isn't limited to standard softmax attention. It appears as a fundamental requirement across various architectural tweaks:

Conclusion

The transition from fixed-length training to "infinite" context inference requires more than just better hardware; it requires a fundamental shift in how we scale the interactions between tokens. By anchoring attention scores to the logarithm of the sequence length, researchers are providing Transformers with the "concentration resilience" needed to navigate massive volumes of data without losing their way. As Alireza Naderi suggests in Mind the Gap, maintaining the "spectral gap" of the attention matrix through proper scaling is what ultimately prevents the model from collapsing into a state where all tokens look the same.

Papers in this version

Showing 60 of 216 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_logarithmic_scaling_for_stable_long_cont_be8959fc,
  title = {Logarithmic Scaling for Stable Long Context Attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/logarithmic-scaling-for-stable-long-context-attention/rdr_be8959fcabde4b279e3443afde42dd80},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_logarithmic_scaling_for_stable_long_cont_be8959fc_v1,
  title = {Logarithmic Scaling for Stable Long Context Attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/logarithmic-scaling-for-stable-long-context-attention/rdr_be8959fcabde4b279e3443afde42dd80/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}