Causal Optimal-Transport Attention: Positional Priors and Length Extrapolation

Ming LianarXiv 2026 · arXiv

TL;DR

This paper introduces a "prefix-consistent" version of Optimal Transport (OT) attention for causal language models, proving it induces a scale-invariant positional prior that improves length extrapolation. While the OT operator significantly outperforms standard softmax, the authors find that its benefits are primarily due to its implicit distance bias, which can be achieved more efficiently using explicit log-distance positional encodings like CABLE.

What Problem Does This Paper Address?

Standard Transformers use softmax attention, which can lead to "rank collapse" and poor performance when evaluating sequences longer than those seen during training (poor length extrapolation). Researchers have previously suggested using Optimal Transport (OT) to "balance" attention, ensuring every key receives a fair share of total attention mass.

However, applying OT to causal language models is tricky for two reasons:

  1. Degeneracy: The only lower-triangular matrix that is both row and column stochastic is the identity matrix, making full balancing useless for causal masks.
  2. Information Leakage: Standard column normalization requires looking at future rows, which leaks labels during training (teacher forcing).

This paper seeks a mathematically sound way to apply OT balancing to causal models and analyzes the "positional prior"—the inherent preference for certain token positions—that this balancing creates.

Methods

The authors propose a prefix-consistent unbalanced OT operator. Instead of balancing against the entire column, it balances against the prefix mass (the attention a key has received from all previous queries in the sequence).

Key technical components include:

  • Damping ($\rho$): A parameter that interpolates between standard softmax ($\rho=0$) and a strongly balanced regime ($\rho \to 1$).
  • Fixed Point Analysis: They derive the exact "limit shape" of the attention pattern when content is ignored. They prove that under bounded scores, attention dilutes: the share of attention given to any fixed-size window vanishes as the context grows.
  • Classification of Encodings: They categorize existing positional encodings (like RoPE, ALiBi, and KERPLE) based on whether they "dilute" or maintain a "bounded far mass" (attention on distant tokens).

Datasets

The models were trained from scratch using:

  • FineWeb-Edu: A high-quality subset of web crawl data (sample-10BT).
  • GPT-2 Tokenizer: Used for processing the text data.
  • Evaluation: Measured by validation loss (nats per token) on sequences 2x to 16x longer than the training length ($L$).

Key Insights and Ideas

  • Scale Invariance: Causal OT balancing naturally induces a scale-invariant prior. As shown in Figure 1, the attention weight depends on the ratio of the key position to the query position ($j/t$), rather than just the absolute distance.
The implicit prior with constant scores. The left panel shows the scale-invariant limit shape $f(s)$ for different balancing strengths $\rho$. The right panel shows how the fixed point follows a power law near the diagonal.
  • The "Far Mass" Connection: The paper establishes a causal link between "far mass"—the amount of attention placed on tokens very far back in the context—and extrapolation loss. If a model's positional encoding allows attention to dilute too much, performance collapses on long sequences.
  • Efficiency vs. Complexity: While the OT operator is effective, exploratory analysis revealed that an explicit, learnable log-distance bias added to softmax can recover most of the gains at only 40% of the computational cost.
  • Summability Boundary: Positional encodings that have "summable" weights (like ALiBi) keep far mass bounded, whereas "non-summable" ones (like RoPE) cause attention to dilute.

Key Results

  • OT vs. Softmax: At 100M parameters, the OT operator ($\rho=0.5$) reduced validation loss at twice the training length by 0.455 nats compared to standard softmax.
  • CABLE Success: CABLE (a specific positional encoding) outperformed RoPE with YaRN scaling on every tested seed and training length.
  • Grouping by Theory: As predicted by their theoretical classification (Proposition 14), models grouped clearly by their "far mass" behavior. As seen in Figure 3, schemes that dilute (like RoPE) show rising loss and far mass, while those with bounded far mass (like ALiBi) remain stable.
Exploratory analysis of far mass (top) and validation loss (bottom) against evaluation length. The results group perfectly according to the theoretical cases of Proposition 14.

Limitations and Future Work

  • Content Correlations: The exact prior derivations assume constant or i.i.d. scores. How real-world token correlations (e.g., specific words always attending to each other) change this shape remains uncharacterized.
  • Scale: Experiments were conducted on models up to 100M parameters. It remains to be seen if these insights hold at the multi-billion parameter scale.
  • Scope: The theory currently only covers prefix-consistent causal attention and does not extend to bidirectional (encoder) or cross-attention mechanisms.
  • Retrieval vs. Loss: The authors note that while low far mass stabilizes loss, it doesn't automatically guarantee that the model can effectively retrieve or copy information from the distant past—it just means the model isn't being "disturbed" by it.

Summary updated

Abstract

Causal masking obstructs the column balancing used by optimal-transport (OT) attention. We study a prefix-consistent unbalanced operator and characterize the positional prior it induces. Under bounded scores, for any fixed balancing strength $ρ<1$, its fixed point dilutes attention: every fixed window receives a vanishing share as the context grows. For constant scores, we derive an exact scale-invariant prior. We then classify positional encodings by whether their distant attention mass remains bounded away from one. Discrete window-mass bounds quantify the finite-length dilution of summable log-distance priors near exponent one. Preregistered experiments with 25M- and 100M-parameter language models establish improved loss extrapolation over softmax; exploratory comparisons favor an explicit distance bias at a fraction of the cost. CABLE outperforms RoPE with evaluation-time YaRN on every seed at three training lengths. Changing only the bias beyond the training length moves long-context loss in the predicted direction on all 30 tested checkpoints. Even with exponents constrained above one, many heads approach the summability boundary during training. A margin selected by prior-only calculations for a specified window, length range and growth tolerance controls distant-mass growth in the tested setting. These results connect loss extrapolation to positional decay over the lengths actually evaluated.
BibTeX
@misc{lian2026causal,
  author = {Ming Lian},
  title = {{Causal Optimal-Transport Attention: Positional Priors and Length Extrapolation}},
  year = {2026},
  eprint = {2611.01472},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01472},
}
Figure