Entropy Invariant Attention Scaling for Long Contexts

15 concepts7 papers2 proposals

When a Transformer model is trained on short sequences and evaluated on much longer ones, its performance often degrades sharply. This failure to generalize is not merely a failure of positional encodings; it is rooted in a fundamental property of the softmax function known as attention dilution or softmax dispersion.

As the input sequence length or image resolution grows, the number of keys in the attention mechanism increases. Because the softmax function normalizes across all keys, distributing the probability mass over a much larger pool of candidates naturally flattens the attention distribution. The model's focus becomes diluted, causing it to lose its ability to distinguish between highly relevant tokens and background noise.

To solve this, a powerful and elegant framework has emerged: length-adaptive temperature scaling. By dynamically adjusting the attention logits based on the sequence length, researchers can preserve the entropy of the attention matrix, ensuring that the model maintains a consistent level of sharpness and focus regardless of the input size.


The Intuition: Why Softmax Dilutes

Standard dot-product attention computes attention weights by scaling query-key inner products by a fixed factor (typically $1/\sqrt{d_k}$) and passing them through a softmax operation:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

This fixed scaling factor assumes that the number of tokens remains relatively constant between training and inference. However, if a model trained on a sequence length of 512 is suddenly evaluated on 8,192 tokens, the softmax denominator must sum over sixteen times as many elements.

If the logits ($QK^T$) follow a similar distribution across lengths, adding thousands of extra keys forces the maximum probability assigned to any single critical token to collapse toward a uniform distribution. The attention matrix becomes uniform, entropy spikes, and the model's capacity to extract specific context vanishes.


The Solution: Length-Adaptive Scaling Laws

To counteract this dispersion, the attention logits must be sharpened as the sequence length grows. This is achieved by introducing a sequence-length adaptive scaling factor—or inverse temperature parameter—that modulates the logits before the softmax operation.

Standard Attention (Long Context):
Logits ───> [ Softmax ] ───> Diluted Weights (Uniform/Blurry)

Length-Adaptive Attention (Long Context):
Logits ───> [ Scale by f(Length) ] ───> [ Softmax ] ───> Sharp Weights (Focused)

The precise mathematical form of this scaling function varies, but it is fundamentally tied to the rate at which entropy expands.

Logarithmic and Square-Root Scaling

Recent theoretical work has pinpointed the exact rate at which this sharpening must occur. Sam Hilton-Jones et al. analyzed attention distributions using Aitchison geometry and proved that to prevent the maximum attention probability from collapsing, the inverse temperature of the softmax must grow at a rate proportional to the square root of the logarithm of the sequence length ($\sqrt{\log L}$).

Similarly, Pavlo Vasylenko et al. demonstrated that scaling the attention temperature based on the logarithm of the sequence length successfully counteracts the natural growth of attention scores, preventing the distribution from becoming too diffuse or overly peaky.

Learnable and Head-Specific Adaptations

Rather than relying solely on rigid closed-form equations, several approaches introduce flexibility by making these scaling factors learnable or context-dependent:

  • Neural Entropy Prediction: Xinyuan Tian et al. proposed Entropy Invariant Extrapolation (EIE), which uses a small auxiliary neural network to predict a dynamic scaling factor. This network explicitly forces the attention matrix to maintain a stable entropy across varying input sizes.
  • Head-Specific Tuning: Not all attention heads behave identically; some focus on local syntax while others track global context. Shaowen Wang et al. introduced head-specific learnable inverse temperature factors, allowing individual heads to modulate their sharpness independently to maximize retrieval accuracy in long contexts.
  • Empirical Ratios: In document-level translation, Zhang Zhuocheng et al. applied a scaling factor based directly on the ratio of the current sequence length to the average training length, stabilizing translation quality when documents exceed the training window.

Cross-Domain Impact: Beyond Text

While length-adaptive scaling is critical for Large Language Models, the exact same mathematical vulnerability applies to any domain where the input dimension varies between training and testing—most notably, computer vision.

When a Vision Transformer (ViT) is trained on small image crops (e.g., $224 \times 224$ pixels) and deployed on high-resolution images or video streams, the number of visual tokens scales quadratically with the image dimensions. This triggers the exact same attention dilution problem.

To enable cross-resolution generalization, vision researchers have adopted entropy-preserving scaling laws:

  • Foveal Vision and Resolution Invariance: Dai Shi utilized dynamic sequence-length scaling factors to maintain entropy invariance during inference, allowing vision models to generalize seamlessly to resolutions significantly higher than those seen during training.
  • Multi-View Stereo: In 3D reconstruction, Chenjie Cao et al. implemented sequence-length adaptive attention scaling to prevent the dilution of cost volumes at high resolutions, preserving the model's ability to focus on fine-grained structural features.
  • Optical Flow Estimation: For dense pixel-matching across video frames, Qiaole Dong et al. applied a resolution-adaptive scaling factor based on the logarithm of the sequence length and image area, enabling models trained on low-resolution crops to maintain precise tracking on high-resolution video.

Summary

Whether processing a massive legal document, a high-definition video, or a complex combinatorial graph, the core challenge remains identical: more data points dilute standard softmax focus. By scaling the attention temperature as a function of sequence length or area, length-adaptive attention scaling preserves the distributional entropy, ensuring that the Transformer's spotlight remains sharp regardless of context size.

Key Foundational Work

MemFlow: Optical Flow Estimation and Prediction with Memory (2024) — contributed 3 concept elements to this direction.
MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo (2024) — contributed 2 concept elements to this direction.
Addressing the Length Bias Challenge in Document-Level Neural Machine Translation (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_logarithmic_temperature_scaling_8253fe7b,
  title = {logarithmic temperature scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/logarithmic-temperature-scaling/rdr_8253fe7bfe02443a90fd76f0de4a0ed9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_logarithmic_temperature_scaling_8253fe7b_v1,
  title = {logarithmic temperature scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/logarithmic-temperature-scaling/rdr_8253fe7bfe02443a90fd76f0de4a0ed9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake