LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models

Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang

Abstract

In recent years, there have been remarkable advancements in the performance of Transformer-based Large Language Models (LLMs) across various domains. As these LLMs are deployed for increasingly complex tasks, they often face the need to conduct longer reasoning processes or understand larger contexts. In these situations, the length generalization failure of LLMs on long sequences becomes more prominent. Most pre-training schemes truncate training sequences to a fixed length. LLMs often struggle to generate fluent and coherent texts, let alone carry out downstream tasks, after longer contexts, even with relative positional encoding designed to cope with this problem. Common solutions such as finetuning on longer corpora often involve daunting hardware and time costs and require careful training process design. To more efficiently leverage the generation capacity of existing LLMs, we theoretically and empirically investigate the main out-of-distribution (OOD) factors contributing to this problem. Inspired by this diagnosis, we propose a simple yet effective solution for on-the-fly length generalization, LM-Infinite. It involves only a $\Lambda$-shaped attention mask (to avoid excessive attended tokens) and a distance limit (to avoid unseen distances) while requiring no parameter updates or learning. We find it applicable to a variety of LLMs using relative-position encoding methods. LM-Infinite is computationally efficient with $O(n)$ time and space, and demonstrates consistent text generation fluency and quality to as long as 32k tokens on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream tasks such as passkey retrieval, it continues to work on inputs much longer than training lengths where vanilla models fail immediately.

The Challenge of Long-Context LLMs

Most modern Large Language Models (LLMs), such as Llama-2 or MPT, are pre-trained on relatively short text segments, typically ranging from 2,000 to 4,000 tokens. This limitation is largely due to the quadratic computational cost of the standard attention mechanism—doubling input length quadruples the required computation.

When these models encounter sequences exceeding their training limit, performance typically collapses. Even models utilizing relative positional encodings, which are theoretically designed to handle varying lengths, fail to generalize. This creates a significant bottleneck for real-world applications requiring the processing of entire books, extensive codebases, or long-running dialogues. LM-Infinite addresses this by identifying the core reasons for this failure and providing a "zero-shot" fix that requires no retraining.

LM-Infinite: A Parameter-Free Solution

LM-Infinite introduces two primary modifications to the attention mechanism that can be applied to off-the-shelf models with minimal code changes:

  1. $\Lambda$-shaped Attention Mask: Standard sliding window attention mechanisms often discard early tokens. However, research shows that the initial tokens in a sequence act as "attention sinks," anchoring the model's internal representations. The $\Lambda$-mask ensures the model always attends to these initial tokens plus a local window of the most recent tokens, effectively bypassing the "middle" tokens that contribute to computational bloat without providing critical structural anchors.
  2. Distance Ceiling: Relative positional encodings like RoPE or ALiBi encounter "unseen distances" during long-context tasks. LM-Infinite caps the relative distance at the maximum value observed during training. This prevents the numerical instability and "explosion" of attention scores that occur when the model attempts to calculate positions far beyond its training distribution.
Conceptual model of relative positional attention in LM-Infinite, distinguishing between absolute and relative position encoding.

Diagnosing the Failure: Why Models Break

The development of LM-Infinite was driven by a rigorous diagnosis of why LLMs fail on long sequences. The authors identified three primary Out-of-Distribution (OOD) factors:

  • The "Attention Sink" Phenomenon: Initial tokens occupy a distinct region in the model's feature space. Discarding them causes hidden states to drift OOD, leading to a collapse in generation quality.
  • Logit Explosion: Theoretically, for models to distinguish between very distant tokens, attention logits must trend toward infinity. This leads to numerical instability.
  • Entropy Explosion: Attending to an ever-increasing number of tokens spreads attention too thin. By limiting the attention window, LM-Infinite keeps attention entropy within the range seen during training.
Empirical diagnosis of OOD factors: logit explosion (a), entropy explosion (b), and the distinct feature space of starting tokens (c).

Performance and Efficiency

LM-Infinite demonstrates remarkable stability and efficiency across various benchmarks:

  • Extreme Generalization: While vanilla Llama-2 fails shortly after 4k tokens, LM-Infinite maintains stable performance on synthetic sequences up to 200 million tokens.
  • Efficiency Gains: On 32k-length sequences, it achieves a 2.7x decoding speedup and a 7.5x reduction in GPU memory usage compared to full-attention models.
  • Downstream Task Success: In Passkey Retrieval ("needle in a haystack") tests, LM-Infinite enabled Llama-2 to achieve 81.2% average accuracy across various lengths, whereas the original model failed immediately (0% accuracy).
  • Competitive with Fine-Tuning: MPT-7B with LM-Infinite performed nearly as well as "MPT-7B-Storywriter," a version specifically fine-tuned for long sequences at significant expense.

Conclusion and Limitations

LM-Infinite provides a robust, "plug-and-play" solution for extending the context window of existing LLMs. However, it has limitations. It requires access to the attention mask, making it inapplicable to closed-source API-only models like GPT-4. Additionally, the $\Lambda$-mask's omission of middle tokens might lead to information loss in certain contexts, though this can be mitigated with more complex selection mechanisms. Future work may involve integrating LM-Infinite directly into the pre-training phase to build more efficient models from the ground up.

Summary updated

Abstract

In recent years, there have been remarkable advancements in the performance of Transformer-based Large Language Models (LLMs) across various domains. As these LLMs are deployed for increasingly complex tasks, they often face the need to conduct longer reasoning processes or understand larger contexts. In these situations, the length generalization failure of LLMs on long sequences becomes more prominent. Most pre-training schemes truncate training sequences to a fixed length. LLMs often struggle to generate fluent and coherent texts, let alone carry out downstream tasks, after longer contexts, even with relative positional encoding designed to cope with this problem. Common solutions such as finetuning on longer corpora often involve daunting hardware and time costs and require careful training process design. To more efficiently leverage the generation capacity of existing LLMs, we theoretically and empirically investigate the main out-of-distribution (OOD) factors contributing to this problem. Inspired by this diagnosis, we propose a simple yet effective solution for on-the-fly length generalization, LM-Infinite. It involves only a $\Lambda$-shaped attention mask (to avoid excessive attended tokens) and a distance limit (to avoid unseen distances) while requiring no parameter updates or learning. We find it applicable to a variety of LLMs using relative-position encoding methods. LM-Infinite is computationally efficient with $O(n)$ time and space, and demonstrates consistent text generation fluency and quality to as long as 32k tokens on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream tasks such as passkey retrieval, it continues to work on inputs much longer than training lengths where vanilla models fail immediately.
BibTeX
@misc{han2023lm,
  author = {Chi Han and Qifan Wang and Wenhan Xiong and Yu Chen and Heng Ji and Sinong Wang},
  title = {{LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models}},
  year = {2023},
  eprint = {2308.16137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.16137},
}
Figure