TL;DR
Standard Transformers struggle with long contexts because the softmax function dilutes attention across thousands of irrelevant tokens. This paper replaces softmax with $\alpha$-entmax to enable exact mathematical sparsity and introduces ASEntmax, an adaptive scaling mechanism that dynamically adjusts attention sharpness based on context length. The result is a massive boost in length extrapolation—up to 1000$\times$ on synthetic tasks—and superior long-context language modeling without sacrificing short-context performance.
What Problem Does This Paper Address?
As researchers scale Transformer context windows to millions of tokens, the standard softmax function becomes a fundamental bottleneck. Because softmax uses an exponential function that never reaches zero, it forces the model to distribute attention mass across every single token in a sequence.
In very long sequences, this density causes three major mathematical pathologies:
- Attention Dispersion: Attention weights inevitably approach a uniform distribution as the context grows, diluting the signal of relevant tokens.
- Representational Collapse: Token representations become highly similar and mathematically indistinguishable in deep layers.
- Over-Squashing: Information from distant inputs is exponentially diluted, causing gradients to vanish across long distances.
While existing solutions modify positional encodings (like RoPE or ALiBi) or scale logits globally, they often require careful tuning and fail to address the root cause: the dense nature of softmax itself.
Methods
The authors propose replacing softmax with $\alpha$-entmax, a differentiable transformation that naturally induces sparsity by assigning exact zeros to irrelevant tokens.
To make this robust across varying sequence lengths, they introduce Adaptive-Scalable Entmax (ASEntmax). Instead of using a fixed sparsity parameter, ASEntmax rescales attention logits per head using a learnable, context-length-dependent inverse temperature schedule:
$$\text{ASEntmax}(z) = \alpha\text{-entmax}((\delta + \beta(\log n)^\gamma)z)$$
Here, $\beta$ and $\gamma$ are learnable, head-specific projections, and $n$ is the sequence length. The inclusion of the $\gamma$ exponent allows the model to dynamically scale or dampen attention sharpness as the context grows.
Additionally, the authors propose NAPE (NoPE + ALiBi), a hybrid positional encoding default where half of the attention heads use ALiBi (inducing a strict recency bias/local window) and the other half use NoPE (No Positional Encoding, allowing purely content-driven long-range retrieval).
Datasets
The authors evaluated their approach on two fronts:
- Synthetic Benchmarks: A suite of length-generalization probing tasks including Multi-query Multi-token Associative Recall (MQMTAR), Max Retrieval, Copy, 2Back, Sort, Reverse, and Flip-Flop.
- Language Modeling: A 420M-parameter LLaMA-3 style model trained on 7B tokens from the DCLM-Edu dataset with a baseline context length of 2,048 tokens.
- Downstream Evaluation: Short-context performance was tested on standard datasets (Lambada, HellaSwag, PIQA, ARC-C, Winogrande, OpenBookQA). Long-context extrapolation was assessed via perplexity on ArXiv/PubMed subsets of The Pile and retrieval accuracy on the RULER benchmark.
Key Insights and Ideas
- Exact Zeros Prevent Leakage: In a softmax distribution, adding a new token strictly decreases the attention mass of all existing tokens. With $\alpha$-entmax, thresholding allows the attention probability of important tokens to remain completely unchanged, regardless of how many irrelevant tokens are added to the context.
- Pruning the Gradient Graph: By restricting the attention distribution to a sparse support size $s \ll n$, $\alpha$-entmax prunes the effective gradient paths through an $L$-layer network from $\mathcal{O}(n^L)$ down to $\mathcal{O}(s^L)$. This concentration of gradient flow directly counteracts over-squashing.
- Heads Need Unique Sparsity Schedules: Empirical analysis showed that different attention heads learn drastically different patterns. The learnable $\gamma$ parameter allows some heads to sharpen their focus in long contexts, while others learn negative $\gamma$ values to spread attention out for memory-heavy tasks like copying.
Key Results
- 1000$\times$ Length Extrapolation: Trained on sequences of just 64 tokens, ASEntmax achieved 95.3% accuracy on the associative recall (MQMTAR) task at a test length of 65,536 tokens, completely outperforming dense baselines.
- Robust Language Modeling: When evaluating the 420M-parameter language model on ArXiv text, ASEntmax maintained a steadily decreasing perplexity trend up to 8$\times$ its training length (16K tokens).
- Superior Needle-in-a-Haystack Retrieval: On the RULER benchmark at 16K tokens (8$\times$ training length), ASEntmax maintained a 97.4% retrieval accuracy on passkey tasks, whereas the standard softmax baseline collapsed entirely to 0.8%.
Limitations and Future Work
- Global Ordering Challenges: While ASEntmax excels at retrieval and copying over massive contexts, tasks requiring precise global sorting (like the Sort task) still fail when extrapolating beyond 4$\times$ the training length, indicating that architectural sparsity alone doesn't fully solve compositional ordering logic.
- Positional Encoding Interactions: The choice of positional encoding heavily influences the shape of the sparse attention graph. For instance, pairing $\alpha$-entmax with RoPE can cause highly fragmented, oscillatory sparsity patterns, meaning care must be taken to design compatible positional priors (like the proposed NAPE hybrid).
Summary updated