Large Language Models (LLMs) like BART and T5 have revolutionized text summarization. However, they face a significant "memory wall" when processing long-form content such as hour-long meeting transcripts or extensive government reports. Most standard models are limited to a context window of 1,024 tokens, whereas a single meeting transcript can easily exceed fifty pages.
Researchers from Penn State, Yale, CMU, and Microsoft Research address this limitation with Summ$^N$, a multi-stage "split-then-summarize" framework. Summ$^N$ allows standard, pretrained LLMs to process documents of arbitrary length by recursively compressing text through multiple stages without requiring modifications to the underlying Transformer architecture.
The Summ$^N$ Framework
The core innovation of Summ$^N$ is its recursive, multi-stage pipeline designed to expand the "receptive field" of pretrained Transformers while maintaining their full representational power.
The framework operates in three primary phases:
- Data Segmentation: The long source text is divided into segments ($S_i$) that fit within the model's maximum input length (e.g., 1,024 tokens).
- Greedy Target Matching: To train the model effectively, the authors developed a ROUGE-based greedy algorithm. This algorithm identifies specific sentences in the gold summary that correspond to each source segment, ensuring the model learns from relevant local context rather than the entire global summary at once.
- Recursive Summarization: In the Coarse Stages, the model generates a summary for each segment. These are concatenated to form the input for the next stage. This process repeats until the concatenated text fits into a single model window. Finally, a Fine-Grained Stage produces the polished final summary.
Summ$^N$ is backbone-agnostic, meaning it can utilize various pretrained models like BART, T5, or PEGASUS as its underlying engine.
Experimental Evaluation
The researchers evaluated Summ$^N$ across several challenging long-text domains:
- Meetings: AMI, ICSI, and QMSum (query-based), which are notoriously difficult due to speaker overlaps and disfluencies.
- TV Scripts: SummScreen, featuring transcripts from soap operas and TV shows.
- Government Reports: GovReport, consisting of large-scale reports from the U.S. Government Accountability Office (GAO).
Results and Key Insights
Summ$^N$ consistently outperformed previous state-of-the-art (SOTA) models across all tested datasets. For instance, on the ICSI meeting dataset, it improved ROUGE-2 and ROUGE-L scores by nearly 2 points over the previous best model, HMNet. On the SummScreen-FD dataset, it achieved a significant jump in ROUGE-1 from 25.90 to 32.48.
Key insights from the study include:
- The Pyramid Effect: Treating summarization as recursive compression allows the model to gain a more global perspective at each stage, effectively "zooming out" on the document.
- Preserving Transformer Integrity: By using standard attention mechanisms, Summ$^N$ leverages the high-quality representations from original pretraining, unlike "efficient" Transformers that sparsify attention.
- Linear Complexity: Summ$^N$ reduces the quadratic time complexity ($O(n^2)$) of standard Transformers to effectively linear ($O(n)$) by processing fixed-size chunks.
- Efficiency: For the GovReport dataset, Summ$^N$ reduced inference time to just 22.8% of the cost compared to a standard quadratic Transformer.
Human evaluations further supported these findings, with annotators rating Summ$^N$ significantly higher in readability and coverage, noting its superior ability to capture the entire narrative arc of long conversations.
Conclusion
Summ$^N$ provides a robust and efficient solution for long-form summarization by bypassing hardware constraints through a multi-stage approach. While the current framework uses separate models for each stage—increasing parameter count—future work may explore parameter sharing and more sophisticated, discourse-aware segmentation to further refine the process.
Summary updated