Abstract
Built upon the Transformer, large language models (LLMs) have captured worldwide attention due to their remarkable abilities. Nevertheless, all Transformer-based models including LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, namely, they cannot perform length extrapolation to handle long sequences, which severely hinders their application in scenarios demanding long input sequences such as legal or scientific documents. Thus, numerous methods have emerged to enhance the length extrapolation of Transformers. Despite the great research efforts, a systematic survey is still lacking. To fill this gap, we delve into these advances in a unified notation from the perspective of positional encoding (PE), as it has been considered the primary factor on length extrapolation. Specifically, we begin with extrapolatable PEs that have dominated this research field. Then, we dive into extrapolation methods based on them, covering position interpolation and randomized position methods. Finally, several challenges and future directions in this area are highlighted.
Introduction
The "Achilles' heel" of the Transformer architecture is its fixed-length training. Because self-attention is permutation-invariant—meaning it does not inherently recognize the order of tokens—positional information must be injected via Positional Encodings (PE). However, if a model is trained on sequences of length $N$, it often fails catastrophically when asked to handle length $N+1$ at inference time. This survey identifies PE as the primary factor in length extrapolation: the ability of a model to generalize from short training sequences to longer inference ones without significant performance degradation or excessive computational cost.
A Taxonomy of Length Extrapolation
Solutions to the length extrapolation problem can be broadly categorized into two eras: the development of inherently extrapolatable PEs and the refinement of these methods for modern LLMs through interpolation and randomization.
1. Extrapolatable PEs (The Foundation)
- Absolute Positional Encodings (APEs): These assign a unique vector to each index. While standard in early models like the original Sinusoidal PE, they struggle to extrapolate because the model never encounters indices beyond the training limit during its learning phase.
- Relative Positional Encodings (RPEs): These focus on the distance between tokens rather than absolute indices. Key examples include ALiBi, which adds a linear penalty to attention scores based on distance, and RoPE (Rotary Positional Encoding), which rotates embeddings in complex space. RPEs are generally superior for extrapolation as they emphasize local relationships.
2. LLM-Era Methods (Refinements)
- Position Interpolation (PI): Instead of extrapolating to unseen indices, PI "squashes" long input sequence indices back into the range seen during training.
- NTK-Aware Interpolation: A sophisticated version of PI that scales different frequency components of the encoding differently, preventing the loss of high-frequency information necessary for distinguishing nearby tokens.
- Randomized PE: Techniques like PoSE introduce random skipping or subsampling during fine-tuning to expose the model to a wider range of relative distances than seen in standard training.
Key Insights and Results
Shift Invariance and Stability
For a model to extrapolate effectively, its PE should ideally be shift-invariant, meaning the relationship between two tokens remains constant regardless of their absolute position in the sequence. Research shows a catastrophic failure of APEs; for instance, Sinusoidal APEs show a massive spike in perplexity (e.g., jumping from ~20 to ~43 on WikiText-103) when the evaluation window exceeds the training window. In contrast, RPE methods like T5-Bias and ALiBi maintain remarkably stable perplexity even when evaluation length is doubled.
The Interpolation Advantage
In the current LLM landscape, it is often more effective to interpolate (downscale) long positions into the trained window than to attempt to teach the model to handle entirely new, larger position indices. While RoPE is the de facto standard for modern LLMs (like Llama), it requires specific interpolation techniques like NTK-aware scaling to handle lengths beyond its training limit effectively.
Benchmarking and Challenges
While benchmarks like LongBench-E and standard language modeling datasets (WikiText-103) provide quantitative metrics, the authors highlight the "Lost in the Middle" phenomenon. Even if a model can technically process 128k tokens, it often struggles to utilize information located in the middle of the document, favoring the beginning and end. Furthermore, popular tests like "Needle In A Haystack" may only measure simple retrieval rather than true long-context reasoning.
Future Directions
Despite progress, several challenges remain:
- Theoretical Grounding: A unified mathematical theory explaining why certain PEs extrapolate better than others is still missing.
- Efficiency: Most methods still face the quadratic complexity of self-attention. Bridging the gap between length extrapolation and linear-time architectures (e.g., Mamba) is a critical next step.
- Utilization: Expanding the context window is only useful if the model can effectively utilize the entire input. Improving long-context utilization remains a primary open challenge.
Summary updated