Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, Ting Liu

TL;DR

Transformers, the backbone of modern LLMs, struggle to generalize beyond their training sequence length, a limitation hindering applications requiring long inputs. This survey systematically categorizes and analyzes positional encoding (PE) methods designed to improve this "length extrapolation" capability, offering a unified perspective on existing solutions and future research directions.

What Problem Does This Paper Address?

Large Language Models (LLMs) based on the Transformer architecture have achieved remarkable success. However, a fundamental limitation persists: they are trained with a fixed maximum input sequence length. This constraint means they often fail to generalize to sequences longer than what they've seen during training, a phenomenon known as "length extrapolation." This inability to handle long sequences severely restricts their application in domains like processing legal documents, scientific literature, or even long-form creative writing. The paper aims to provide a comprehensive overview of the research efforts dedicated to overcoming this challenge, focusing on the crucial role of positional encoding (PE) techniques.

Methods

This paper presents a survey of methods for improving length extrapolation in Transformers, primarily by analyzing positional encoding (PE) techniques. The authors categorize PEs into two main types:

  • Absolute Positional Encodings (APEs): These assign a unique representation to each position. While foundational (like sinusoidal embeddings), they often struggle with extrapolation. The survey discusses APE variants that attempt to integrate "shift invariance" or enhance "smoothness" of position representations.
  • Relative Positional Encodings (RPEs): These encode the relative distance between tokens, which is argued to be more aligned with natural language and inherently better for extrapolation due to their shift-invariance. The survey details RPEs that modify attention scores directly, including the popular ROPE (Rotary Position Embedding) family and T5-bias family methods.

Beyond PE, the survey also covers two main categories of extrapolation methods:

  • Position Interpolation: Techniques that scale down position indices or modify the base of PE (like ROPE) during inference to fit within the original training context window.
  • Randomized Positional Encoding: Methods that expose models to a wider range of positions during training, either by randomly selecting subsets or by using skipping biases, to improve generalization to unseen lengths.

Datasets

The paper references various datasets used in the literature for evaluating length extrapolation, including:

  • LongBench-E: Used for evaluating generation tasks across different LLMs.
  • WikiText-103, OpenWebText2, ArXiv: Used for language modeling evaluations.
  • Synthetic tasks: Mentioned for analyzing length generalization behavior (e.g., arithmetic, deductive reasoning).

The authors also discuss the need for standardized benchmarks for length extrapolation and propose preliminary thoughts on their construction in Appendix A.3.

Key Insights and Ideas

  • PE is Central: The paper strongly emphasizes that positional encoding is the primary factor influencing a Transformer's ability to extrapolate to longer sequences.
  • RPEs Outperform APEs for Extrapolation: Relative Positional Encodings (RPEs) generally show better performance for length extrapolation compared to Absolute Positional Encodings (APEs), largely due to their inherent shift-invariance.
  • ROPE is the De Facto Standard: Rotary Position Embeddings (ROPE) have become the dominant PE in recent LLMs due to their strong in-distribution performance, despite initial concerns about extrapolation.
  • Interpolation vs. Randomization: Position interpolation methods are generally "plug-and-play" and require minimal fine-tuning, making them highly appealing. Randomized PEs, while effective, often necessitate further fine-tuning.
  • Perplexity Isn't Everything: The survey highlights that while bias-based RPEs might achieve lower perplexity (indicating better language modeling), embedding-based methods like ROPE might offer better overall performance on downstream tasks, suggesting perplexity alone is insufficient for evaluation.
  • "Lost in the Middle" Problem: Even with extended context windows, LLMs struggle to effectively utilize information in the middle of long sequences, indicating that simply increasing context length isn't enough.

Key Results

The paper is a survey, so it doesn't present novel experimental results but rather synthesizes findings from existing research. Key findings highlighted include:

  • Consistent Degradation: Across various models and tasks, performance consistently degrades when sequences extend beyond the original training context window, underscoring the necessity of length extrapolation research.
  • RPEs Maintain Performance: RPE-based models tend to show a gradual performance decline rather than an abrupt drop when extrapolating, thanks to their shift-invariance and decay properties.
  • Empirical Comparisons: Tables 2 and 3 present comparative results from various studies, showing that RPEs (especially T5 bias and ALiBi) often outperform APEs (like Sinusoidal) on language modeling tasks, both within and beyond the context window. ROPE also shows competitive extrapolation capabilities.

Limitations and Future Work

Limitations:

  • Lack of Standardized Benchmarks: The authors note the absence of standardized benchmarks and evaluation methods, making fine-grained empirical analysis challenging.
  • Focus on Real-World Scenarios: The survey primarily focuses on length extrapolation for extending LLM context windows in real-world scenarios, with only brief discussions on studies analyzing length generalization in synthetic tasks.
  • PE is Crucial, Not Solely Sufficient: While PE is critical, the paper acknowledges that length generalization requires a systematic design where PE is a component, but not the only one.

Future Work:

  • Develop Standardized Benchmarks: The paper calls for dedicated benchmarks and evaluation methods to further advance the field.
  • Enhance Explainability: A deeper theoretical understanding of why and when Transformers extrapolate is needed, moving beyond heuristics like "decaying-with-distance."
  • Optimize Long Context Utilization: Research should focus not just on expanding context windows but also on effectively utilizing the information within them, addressing issues like the "lost in the middle" problem.
  • Investigate Novel Architectures: While Transformers are dominant, the paper mentions the emergence of alternative architectures (e.g., S4, RWKV, Hyena, Mamba) and suggests further evaluation of their Transformer-beating potential.

Summary updated

Abstract

Built upon the Transformer, large language models (LLMs) have captured worldwide attention due to their remarkable abilities. Nevertheless, all Transformer-based models including LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, namely, they can not perform **length extrapolation** to handle long sequences. Thus, numerous methods have emerged to enhance the length extrapolation of Transformers. Despite the great research efforts, a systematic survey is still lacking. To fill this gap, we delve into these advances in a unified notation from the perspective of positional encoding (PE), as it has been considered the primary factor on length extrapolation. Specifically, we begin with extrapolatable PEs that have dominated this research field. Then, we dive into extrapolation methods based on them, covering position interpolation and randomized position methods. Finally, several challenges and future directions in this area are highlighted. Through this survey, We aim to enable the reader to gain a deep understanding of existing methods and provide stimuli for future research.
BibTeX
@inproceedings{zhao2024length,
  author = {Liang Zhao and Xiachong Feng and Xiaocheng Feng and Weihong Zhong and Dongliang Xu and Qing Yang and Hongtao Liu and Bing Qin and Ting Liu},
  title = {{Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.582},
}
Figure