Standard Transformer models like BERT are famously "short-sighted." Because the computational cost of self-attention grows quadratically with the number of tokens, most models are capped at a 512-token limit—roughly the length of a single page. When faced with a 100-page legal brief or a massive clinical history, these models are forced to either truncate the text, losing vital information, or suffer from prohibitive memory costs.
To solve this, researchers have converged on a shared strategy: Hierarchical Encoding. Instead of treating a document as one flat sequence of thousands of tokens, these models treat it as a nested structure of sentences, paragraphs, or "chunks."
The Two-Tier Architecture
The intuition behind hierarchical models is that human reading is multi-level. We don't maintain a simultaneous mental map of every word in a book; instead, we understand a sentence, compress it into a semantic idea, and then relate that idea to the previous paragraph.
In research like NextLevelBERT, Tamara Czinczoll and colleagues implement this by segmenting documents into fixed-size chunks. This creates a two-tier system:
- The Local Encoder: A standard Transformer processes individual chunks (e.g., 512 tokens each). This captures local syntax and immediate context.
- The Global Aggregator: The output of the local encoder—often a single "summary" vector per chunk—is passed to a second, higher-level architecture.
This secondary layer can take many forms. Snehal Khandve and Vedangi Wagh explored using LSTMs or CNNs as aggregators in Hierarchical Neural Network Approaches for Long Document Classification. However, more recent work like the LCD Benchmark suggests that using a second Transformer as the aggregator is more effective, as it allows different segments of a document to "attend" to each other regardless of distance.
Semantic Compression and Efficiency
The primary benefit of this approach is a massive reduction in sequence length. If a document has 10,000 tokens, a standard Transformer requires $10,000^2$ attention operations. By chunking the text into twenty 500-token blocks, the model only needs to perform twenty small $500^2$ operations plus one $20^2$ operation at the global level.
This efficiency allows models to process hundreds of thousands of words. In ChuLo: Chunk-Level Key Information Representation for Long Document Processing, Caren Han and Danni Li demonstrate that by weighting these chunks with keyphrase extraction, models can maintain high performance on complex document understanding without the quadratic "tax" of full-sequence attention.
Capturing Document Structure
Hierarchical models do more than just save memory; they align with the natural structure of specialized domains. In the legal field, for instance, Nikolaos Aletras and Leslie Barrett note that legal documents have specific rhetorical roles—like "findings of fact" or "rulings."
By using architectures like BERTSUM, developed by Yang Liu and Mirella Lapata in Text Summarization with Pretrained Encoders, models can insert multiple classification tokens to generate distinct representations for every sentence. This allows the model to "see" the document's skeleton, making it much better at tasks like Court Judgment Prediction or Narrative Incoherence Detection.
Solving the Position Problem
A major hurdle in long-context modeling is "positional encoding." Standard Transformers use a simple counter (1, 2, 3...) to know where a word is. But in a hierarchical model, a word has two positions: its place in the sentence and the sentence's place in the book.
Zhenyu He and colleagues addressed this in Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation. Their BiPE framework uses "Bilevel Positional Encoding," which separates intra-segment and inter-segment positions. This allows a model trained on short texts to "extrapolate" to much longer documents because the local structure (the length of a sentence) remains constant even as the global structure (the number of sentences) grows.
Why It Works: The Intuition of Abstraction
Hierarchical encoding works because it forces the model to perform semantic abstraction. By the time the information reaches the top-level Transformer, the "noise" of individual tokens has been filtered out into dense, semantic embeddings.
Whether it is Shenghuan Sun at UCSF using hierarchical BERT to analyze long-term social histories of cancer patients, or Linyi Yang using NumHTML to process hour-long financial earnings calls, the principle remains the same: to understand the whole, you must first summarize the parts.