Effective Long-Context Scaling of Foundation Models

Jingyu Liu, Wenhan Xiong, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma

Introduction

Large Language Models (LLMs) have transformed artificial intelligence, yet they frequently encounter a "context wall." Standard models like Llama 2 are typically limited to a 4,096-token window, which restricts their ability to process extensive documents, complex codebases, or long-form narratives. Meta researchers have introduced Llama 2 Long, a series of models with a context window extended to 32,768 tokens. This was achieved through a streamlined recipe of continual pretraining and a targeted modification to positional encodings, resulting in models that excel in long-context scenarios while simultaneously improving performance on standard short-context benchmarks.

The Challenge of Long Context

Simply expanding the context window during inference is insufficient; the model's positional encodings often fail to generalize to longer distances, leading to a breakdown in coherence. Furthermore, many attempts to extend context windows suffer from a "performance tax," where the model's general intelligence—measured by tasks like math or coding—regresses. Meta's objective was to find a scaling recipe that provides a massive context window while enhancing overall model quality.

Methods: Continual Pretraining and RoPE ABF

The researchers utilized three core strategies to develop Llama 2 Long:

  1. Continual Pretraining: Rather than training from scratch, the team started with existing Llama 2 checkpoints and performed additional training on 400 billion tokens. They employed a curriculum that gradually increased sequence lengths, demonstrating that "learning" long context is more efficient than starting with it.
  2. RoPE ABF (Adjusted Base Frequency): Llama 2 utilizes Rotary Positional Encodings (RoPE). To accommodate longer distances between tokens, the researchers increased the base frequency $b$ from 10,000 to 500,000. This modification reduces the decay of attention scores for distant tokens, allowing the model to maintain focus across the entire 32k window. This approach was found to be superior to simple position interpolation as it preserves higher granularity for token positions.
  3. Synthetic Instruction Tuning: To develop a "Chat" version without the high cost of human-annotated long documents, they used a self-instruct approach, where Llama 2 Chat generated synthetic QA pairs from long text segments.

Context as a Fundamental Scaling Axis

A pivotal discovery of this research is that validation loss follows a predictable power-law relationship with context length. This suggests that context length is not merely a feature but a fundamental scaling axis for model intelligence, comparable to parameter count or training data volume.

Validation loss vs. context length across model scales.

Interestingly, ablation studies revealed that the volume of long texts in the training set was less critical than the correct modification of positional encodings. High-quality data proved more important than sheer length, and the model could learn to handle long contexts even when trained on predominantly shorter documents.

Results: Breaking the Performance Tax

Llama 2 Long demonstrates that extending context does not have to come at the cost of general performance. In fact, the Llama 2 Long 70B variant outperformed gpt-3.5-turbo-16k on 7 out of 10 tasks in the ZeroSCROLLS benchmark.

Crucially, the models showed significant improvements on standard short-context benchmarks. For instance, the 70B model's coding score (HumanEval) rose from 37.4 to 39.9, and its math score (GSM8K) improved from 35.2 to 41.3 compared to the original Llama 2.

Comparison of Llama 2 and Llama 2 Long on standard benchmarks.

Limitations and Future Work

Despite these advancements, several challenges remain. The Llama tokenizer is approximately 10% less efficient than proprietary competitors like GPT-4, particularly with whitespace in code. Additionally, while the model can process more information, it still faces issues with hallucinations in high-density contexts. Future work may involve more sophisticated Reinforcement Learning from Human Feedback (RLHF) specifically tailored for long-form reasoning to further unlock the potential of these models.

Summary updated

Abstract

We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks. Alongside these results, we provide an in-depth analysis on the individual components of our method. We delve into Llama's position encodings and discuss its limitation in modeling long dependencies. We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.
BibTeX
@inproceedings{liu2024effective,
  author = {Jingyu Liu and Wenhan Xiong and Igor Molybog and Hejia Zhang and Prajjwal Bhargava and Rui Hou and Louis Martin and Rashi Rungta and Karthik Abinav Sankararaman and Barlas Oguz and Madian Khabsa and Han Fang and Yashar Mehdad and Sharan Narang and Kshitiz Malik and Angela Fan and Shruti Bhosale and Sergey Edunov and Mike Lewis and Sinong Wang and Hao Ma},
  title = {{Effective Long-Context Scaling of Foundation Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.260},
}
Figure