Simple Local Attentions Remain Competitive for Long-Context Tasks

Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, Yashar Mehdad

Introduction

Standard Transformer models scale quadratically with sequence length, making them prohibitively expensive for long documents. In response, the NLP community has proposed numerous "efficient attention" variants, including learnable sparse patterns (e.g., Reformer, Sinkhorn), low-rank/kernel approximations (e.g., Linformer, Performer), and fixed local patterns (e.g., Longformer, BigBird).

However, evaluating these models is difficult due to inconsistent benchmarks and pretraining setups. Many models are tested on synthetic benchmarks like Long Range Arena (LRA) or initialized differently, making it hard for practitioners to identify the best architecture for real-world "pretrain-and-finetune" long-context NLP tasks. This paper provides a large-scale, controlled comparison to address this gap.

Methods

The authors implemented several classes of efficient attention within a unified framework (xformers) to ensure a fair comparison:

  1. Learnable sparse patterns: Sinkhorn, LSH (Reformer)
  2. Low-rank/kernel approximations: Linformer, Nyström, Performer
  3. Hybrid attention: Long-Short
  4. Fixed local patterns: Sliding Window, Blockwise Local Window
Comparison of Local Window and Blockwise Local Window attention patterns.

Instead of relying solely on synthetic benchmarks, the researchers pretrained RoBERTa-large scale models from scratch for all variants using a sequence length of 4,096 tokens. They strictly controlled for memory usage and batch size. Finally, they ablated the winning local attention models to evaluate the necessity of standard design choices, such as overlapping attention windows and global tokens.

Datasets

The evaluation utilized several benchmarks:

  • Synthetic Benchmark: A text-based subset of the Long Range Arena (LRA) benchmark, including ListOps, IMDB sentiment, and text matching.
  • Pretraining: A custom long-document corpus compiled from Stories, RealNews, Books, and English Wikipedia.
  • Downstream Tasks: TriviaQA (extractive QA), Hyperpartisan (classification), and NQ Doc Retrieval.

Key Insights and Results

The Limitations of LRA

A primary finding is that LRA is a misleading benchmark for real-world NLP performance. While almost all attention variants achieved similar performance on LRA (around 60-61% average accuracy), their performance diverged significantly during large-scale pretraining and downstream finetuning. Models that appeared competitive on LRA, such as Performer, failed to perform well on actual NLP tasks.

Performance of various attention models on the Long Range Arena (LRA) benchmark.

Superiority of Simple Local Attention

Fixed local attention, where tokens attend only to immediate neighbors, consistently outperformed complex mathematically sophisticated approximations across real-world tasks. For instance, on TriviaQA, the simple Blockwise Local Window model achieved a 72.9 F1 score, significantly higher than Performer (31.9 F1) and Linformer (65.2 F1).

Efficiency Gains from Disjoint Windows

Standard local attention models often use overlapping blocks to allow information flow across boundaries. The authors demonstrated that this overlap slightly improves pretraining perplexity but does not benefit downstream performance. By removing the overlapping requirement, they created a disjoint local attention model that matched Longformer's performance on TriviaQA while offering a >2x speedup in processing speed (14.8k vs 6.6k words/sec).

The Role of Global Tokens

While overlapping windows are optional, the use of "global tokens" (e.g., the [CLS] token) that can attend to the entire sequence remains crucial for downstream success.

Conclusion and Future Work

This research demonstrates that simple local attention mechanisms remain highly competitive and efficient for long-context tasks. The study focused on encoder-only models; future work will explore whether these findings extend to autoregressive decoder-only architectures. Additionally, the authors suggest that "staged training"—gradually increasing context length during pretraining—could further optimize Masked Language Modeling (MLM).

Summary updated

BibTeX
@inproceedings{xiong2022simple,
  author = {Wenhan Xiong and Barlas Oguz and Anchit Gupta and Xilun Chen and Diana Liskovich and Omer Levy and Scott Yih and Yashar Mehdad},
  title = {{Simple Local Attentions Remain Competitive for Long-Context Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.144},
}
Figure