Introduction
Standard Transformer models scale quadratically with sequence length, making them prohibitively expensive for long documents. In response, the NLP community has proposed numerous "efficient attention" variants, including learnable sparse patterns (e.g., Reformer, Sinkhorn), low-rank/kernel approximations (e.g., Linformer, Performer), and fixed local patterns (e.g., Longformer, BigBird).
However, evaluating these models is difficult due to inconsistent benchmarks and pretraining setups. Many models are tested on synthetic benchmarks like Long Range Arena (LRA) or initialized differently, making it hard for practitioners to identify the best architecture for real-world "pretrain-and-finetune" long-context NLP tasks. This paper provides a large-scale, controlled comparison to address this gap.
Methods
The authors implemented several classes of efficient attention within a unified framework (xformers) to ensure a fair comparison:
- Learnable sparse patterns: Sinkhorn, LSH (Reformer)
- Low-rank/kernel approximations: Linformer, Nyström, Performer
- Hybrid attention: Long-Short
- Fixed local patterns: Sliding Window, Blockwise Local Window
Instead of relying solely on synthetic benchmarks, the researchers pretrained RoBERTa-large scale models from scratch for all variants using a sequence length of 4,096 tokens. They strictly controlled for memory usage and batch size. Finally, they ablated the winning local attention models to evaluate the necessity of standard design choices, such as overlapping attention windows and global tokens.
Datasets
The evaluation utilized several benchmarks:
- Synthetic Benchmark: A text-based subset of the Long Range Arena (LRA) benchmark, including ListOps, IMDB sentiment, and text matching.
- Pretraining: A custom long-document corpus compiled from Stories, RealNews, Books, and English Wikipedia.
- Downstream Tasks: TriviaQA (extractive QA), Hyperpartisan (classification), and NQ Doc Retrieval.
Key Insights and Results
The Limitations of LRA
A primary finding is that LRA is a misleading benchmark for real-world NLP performance. While almost all attention variants achieved similar performance on LRA (around 60-61% average accuracy), their performance diverged significantly during large-scale pretraining and downstream finetuning. Models that appeared competitive on LRA, such as Performer, failed to perform well on actual NLP tasks.
Superiority of Simple Local Attention
Fixed local attention, where tokens attend only to immediate neighbors, consistently outperformed complex mathematically sophisticated approximations across real-world tasks. For instance, on TriviaQA, the simple Blockwise Local Window model achieved a 72.9 F1 score, significantly higher than Performer (31.9 F1) and Linformer (65.2 F1).
Efficiency Gains from Disjoint Windows
Standard local attention models often use overlapping blocks to allow information flow across boundaries. The authors demonstrated that this overlap slightly improves pretraining perplexity but does not benefit downstream performance. By removing the overlapping requirement, they created a disjoint local attention model that matched Longformer's performance on TriviaQA while offering a >2x speedup in processing speed (14.8k vs 6.6k words/sec).
The Role of Global Tokens
While overlapping windows are optional, the use of "global tokens" (e.g., the [CLS] token) that can attend to the entire sequence remains crucial for downstream success.
Conclusion and Future Work
This research demonstrates that simple local attention mechanisms remain highly competitive and efficient for long-context tasks. The study focused on encoder-only models; future work will explore whether these findings extend to autoregressive decoder-only architectures. Additionally, the authors suggest that "staged training"—gradually increasing context length during pretraining—could further optimize Masked Language Modeling (MLM).
Summary updated