Beyond Paragraphs: NLP for Long Sequences

Iz Beltagy, Arman Cohan, Hannaneh Hajishirzi, Sewon Min, Matthew E. Peters2021 · DOI

Abstract

Iz Beltagy, Arman Cohan, Hannaneh Hajishirzi, Sewon Min, Matthew E. Peters. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials.

2021.

Introduction

Most modern Natural Language Processing (NLP) is effectively "short-sighted." Despite the revolutionary success of Transformer-based models like BERT and RoBERTa, these architectures are typically capped at a sequence length of 512 tokens. This limitation is not arbitrary; it is a direct consequence of the Self-Attention mechanism, which possesses a computational and memory complexity of $O(N^2)$.

In practical terms, this "quadratic wall" means that doubling the length of an input text quadruples the resources required to process it. This makes the direct application of standard Transformers to entire scientific papers, legal contracts, or books computationally prohibitive. Historically, researchers bypassed this by "chunking" text into smaller, independent pieces. However, this approach sacrifices long-distance context—the ability for a model to connect a pronoun on page 50 to a character introduced on page 1. This paper outlines a roadmap for moving "Beyond Paragraphs" by adopting more efficient architectures and engineering strategies.

Overcoming the Quadratic Bottleneck

The tutorial categorizes the evolution of long-sequence processing into three primary domains: established heuristics, efficient architectural innovations, and engineering optimizations.

1. Established Heuristics

Before the advent of specialized long-sequence Transformers, several strategies were used to mitigate the length constraint:

  • Chunking: Splitting text into segments and processing them independently. While simple, it suffers from "context fragmentation."
  • Hierarchical Models: Processing individual sentences or paragraphs first, then aggregating those representations into a document-level view using a secondary layer.
  • Retrieval-based Methods: Using a lightweight model to identify the most relevant snippets of a document and feeding only those high-value segments into the primary model.
  • Graph-based Approaches: Representing text snippets as nodes in a graph and using Graph Neural Networks (GNNs) to pass information across the document structure.

2. Efficient Transformer Architectures

The core of contemporary research focuses on approximating the $O(N^2)$ attention mechanism to achieve near-linear complexity. Key models include:

  • Transformer-XL: Introduces recurrence to maintain state across segments, allowing the model to look beyond a single fixed window.
  • Longformer: Utilizes a "sliding window" attention for local context combined with specific "global" attention tokens for critical anchors (like the [CLS] token or question tokens).
  • Reformer: Employs locality-sensitive hashing (LSH) to group similar tokens together, reducing the number of comparisons needed.
  • Linformer: Uses low-rank projection to reduce the size of the attention matrix, effectively compressing the complexity.

3. Engineering Optimizations

Beyond architectural changes, fitting long sequences into GPU memory requires specific engineering tricks. Techniques such as gradient checkpointing (which trades extra computation for reduced memory footprint) and gradient accumulation allow researchers to train on sequences that would otherwise exceed hardware limits.

Benchmarking Long-Sequence Models

Evaluating these models requires tasks that demand document-level understanding rather than simple sentence-level sentiment or syntax. Critical benchmarks include:

  • Multi-hop Question Answering: Tasks like HotpotQA and Wikihop, where the answer must be synthesized from multiple disparate parts of a document.
  • Information Extraction: Document-level relation extraction and coreference resolution (e.g., Pre-Co or OntoNotes).
  • Long-form Summarization: Distilling thousands of words into a concise summary, a task that inherently requires a global view of the text.

Key Insights for Practitioners

One of the most practical takeaways from the tutorial is the "Warm-Start" Strategy. Researchers do not necessarily need to pretrain a long-sequence model from scratch. Instead, one can take a model already trained on short sequences (like RoBERTa) and adapt its position embeddings to handle longer inputs. With a relatively small amount of additional pretraining, the model can learn to utilize the expanded context window.

Furthermore, the success of models like Longformer suggests that a Local vs. Global Balance is essential. Most tokens only need to interact with their immediate neighbors to capture local semantics, while a few "anchor" tokens require a global view to maintain the document's overall structure.

Future Directions and Challenges

While the field has moved from processing 512 tokens to tens of thousands, several hurdles remain:

  • The Output Bottleneck: While we have become efficient at reading (encoding) long sequences, generating (decoding) long sequences remains slow and computationally expensive.
  • Pretraining Objectives: Most models are still pretrained on short text snippets. There is a growing need for pretraining objectives specifically designed to teach models document-level reasoning.
  • Retrieval vs. Reading: The debate continues over whether it is more efficient to build models that "read everything" or to refine two-stage "retrieve-then-read" systems. The optimal solution likely depends on the specific requirements of the task at hand.

Summary updated

BibTeX
@misc{beltagy2021beyond,
  author = {Iz Beltagy and Arman Cohan and Hannaneh Hajishirzi and Sewon Min and Matthew E. Peters},
  title = {{Beyond Paragraphs: NLP for Long Sequences}},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-tutorials.5},
}
Figure