When training Transformer models, computational constraints usually force researchers to choose a fixed maximum sequence length—such as 512 or 2048 tokens. However, at inference time, we often want these models to process much longer inputs, whether it is an entire book, an extended user history, or a massive genomic sequence.
Standard positional encodings fail catastrophically when a model is evaluated on sequences longer than those seen during training. Attention with Linear Biases (ALiBi) solves this problem by replacing learned or functional position embeddings with a simple, non-learned penalty applied directly to the attention matrix. This architectural shift enables true "train short, test long" length extrapolation.
The Extrapolation Wall
To understand why linear biases work, we must first look at why traditional positional encodings break down. In standard self-attention, the similarity between queries ($Q$) and keys ($K$) is computed via a dot product:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
Because the dot product itself is permutation-invariant, Transformers require a mechanism to perceive token order. Traditional approaches inject this spatial awareness by modifying the query and key vectors before the dot product:
- Absolute Positional Encodings: A unique vector is added to each token's embedding based on its exact index. If a model trained on sequences of length 512 encounters a token at position 1024 during inference, it lacks a trained embedding for that position.
- Rotary Positional Encodings (RoPE): Queries and keys are rotated in a complex plane by an angle proportional to their position. While elegant, when the distance between tokens grows larger than the training window, the phase shifts introduce out-of-distribution values that cause the attention scores to degrade.
When standard models encounter these unfamiliar positions or distances, the attention matrix becomes unstable, and performance drops precipitously.
The Mechanism: Penalizing Distance Directly
Introduced by Ofir Press et al. in Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, Attention with Linear Biases bypasses the token representation layer entirely. Instead of adding positional metadata to $Q$ and $K$, the authors introduced a static, non-learned linear penalty directly to the attention scores after the dot product but before the softmax operation.
For a causal language model, the attention score between a query at position $i$ and a key at position $j$ is modified as follows:
$$\text{Score}_{i,j} = q_i k_j^T - m \cdot (i - j)$$
where $i \ge j$, and $m$ is a constant, non-learned geometric slope specific to each attention head. For bidirectional models, such as encoders used in masked language modeling, the penalty is applied symmetrically using the absolute relative distance, $-m \cdot |i - j|$, as explored by Minchul Lee et al. in LittleBird: Efficient Faster & Longer Transformer for Question Answering.
Traditional Attention:
Query (pos 5) ──┐
├─► [ Dot Product ] ─► [ Softmax ] ─► Attention Weights
Key (pos 2) ──┘
Attention with Linear Biases:
Query (pos 5) ──┐
├─► [ Dot Product ] ───┐
Key (pos 2) ──┘ │
▼
Distance: |5 - 2| = 3 ──► [ × Slope m ] ──► [ Subtract Penalty ] ─► [ Softmax ]
Why It Works: The Softmax Shield
The core intuition behind why linear biases extrapolate so robustly lies in how the penalty interacts with the softmax function.
As the relative distance between a query and a key increases, the penalty $-m \cdot |i - j|$ grows linearly more negative. Because softmax exponentiates its inputs ($e^x$), a linear decrease in the input score translates to an exponential decay in the final attention weight.
This creates three powerful inductive biases:
- Locality Preference: The model naturally prioritizes nearby context, which aligns with the statistical properties of most sequential data.
- Immunity to Scale: When the sequence length scales from 512 to 4096, the model never encounters an "out-of-bounds" feature vector or an unseen phase angle. The mathematical relationship between a query and a key separated by 10 tokens is identical whether those tokens reside at the beginning of a short sequence or the end of an ultra-long sequence.
- The Softmax Shield: Distant tokens—those beyond the training context window—receive massive negative penalties. By the time they pass through the softmax function, their attention weights are driven safely to zero. This prevents un-extrapolated, long-range relationships from corrupting or dominating the attention distribution.
To ensure the model can still capture long-range dependencies when necessary, the slope $m$ varies across attention heads. For instance, in an 8-head architecture, the slopes might be set as a geometric progression: $\frac{1}{2^1}, \frac{1}{2^2}, \dots, \frac{1}{2^8}$. Heads with steep slopes (e.g., $m = 0.5$) focus strictly on immediate neighbors, while heads with very shallow slopes (e.g., $m = 0.0039$) act as global aggregators, penalizing distance lightly and allowing information to flow across thousands of tokens.
Cross-Domain Adaptations
Because linear biases decouple sequence length from parameter learning, the architecture has been widely adopted across diverse domains where sequence lengths vary drastically between training and deployment.
Natural Language Processing at Scale
In large-scale language modeling, linear biases eliminate the need to train on massive context windows from day one, saving substantial compute. Teven Le Scao et al. demonstrated the utility of this approach in What Language Model to Train if You Have One Million GPU Hours?, selecting linear biases to maximize zero-shot performance and length extrapolation for the 176-billion parameter BLOOM model. This strategy has been mirrored in regional foundation models like FinGPT (Risto Luukkonen et al., FinGPT: Large Generative Models for a Small Language) and Poro 34B (Jonathan Burdge et al., Poro 34B and the Blessing of Multilinguality). Similarly, Jacob P. Portes et al. integrated linear biases into MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining to optimize training efficiency while maintaining downstream spatial awareness.
Genomics and Bioinformatics
Biological sequences like DNA and RNA are inherently long-range, often spanning hundreds of thousands of base pairs. Traditional sequence-splitting heuristics break critical long-range biological interactions.
To solve this, Zhihan Zhou et al. utilized linear attention biases in DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes, showing that a relative distance penalty captures genomic dependencies far more effectively than learned embeddings. This zero-shot extrapolation capability has become a cornerstone for processing full-length transcripts and structural tasks in genomic foundation models, as seen in dnaGrinder (Qihang Zhao et al., dnaGrinder: a lightweight and high-capacity genomic foundation model), mRNABERT (Ying Xiong et al., mRNABERT: advancing mRNA sequence design with a universal language model and comprehensive dataset), and the BEACON benchmark (Yuchen Ren et al., BEACON: Benchmark for Comprehensive RNA Tasks and Language Models).
Behavioral Logs, Pathology, and Education
The "train short, test long" paradigm extends naturally to other non-text modalities:
- User Modeling: In General-Purpose User Modeling with Behavioral Logs: A Snapchat Case Study, Qixiang Fang et al. applied linear biases to interaction logs, allowing models to seamlessly ingest extended user histories without requiring retraining on larger context windows.
- Digital Pathology: Anurag Jayant Vaidya et al. incorporated linear biases into Multimodal Whole Slide Foundation Model for Pathology, enabling a model trained on smaller, patch-level image regions to scale up to gigapixel-scale whole-slide images during inference.
- Knowledge Tracing: In educational data mining, Xueyi Li et al. demonstrated in Enhancing Length Generalization for Attention Based Knowledge Tracing Models with Linear Biases that a non-learned distance penalty maintains stable performance when tracking a student's learning history over intervals that far exceed the training limits.
By shifting the responsibility of position encoding from learned vector spaces to a static mathematical penalty, linear biases provide a robust inductive bias that aligns perfectly with the mechanics of self-attention, unlocking efficient context extension across domains.
Key Foundational Work
What Language Model to Train if You Have One Million GPU Hours? (2022) — contributed 2 concept elements to this direction.
MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining (2023) — contributed 1 concept elements to this direction.
LittleBird: Efficient Faster & Longer Transformer for Question Answering (2022) — contributed 1 concept elements to this direction.