SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents

Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Hassan Awadallah, Dragomir R. Radev, Rui Zhang

Large Language Models (LLMs) like BART and T5 have revolutionized text summarization. However, they face a significant "memory wall" when processing long-form content such as hour-long meeting transcripts or extensive government reports. Most standard models are limited to a context window of 1,024 tokens, whereas a single meeting transcript can easily exceed fifty pages.

Researchers from Penn State, Yale, CMU, and Microsoft Research address this limitation with Summ$^N$, a multi-stage "split-then-summarize" framework. Summ$^N$ allows standard, pretrained LLMs to process documents of arbitrary length by recursively compressing text through multiple stages without requiring modifications to the underlying Transformer architecture.

The Summ$^N$ Framework

The core innovation of Summ$^N$ is its recursive, multi-stage pipeline designed to expand the "receptive field" of pretrained Transformers while maintaining their full representational power.

The multi-stage architecture of Summ$^N$, illustrating data segmentation, coarse summary generation, and fine-grained summary generation.

The framework operates in three primary phases:

  1. Data Segmentation: The long source text is divided into segments ($S_i$) that fit within the model's maximum input length (e.g., 1,024 tokens).
  2. Greedy Target Matching: To train the model effectively, the authors developed a ROUGE-based greedy algorithm. This algorithm identifies specific sentences in the gold summary that correspond to each source segment, ensuring the model learns from relevant local context rather than the entire global summary at once.
  3. Recursive Summarization: In the Coarse Stages, the model generates a summary for each segment. These are concatenated to form the input for the next stage. This process repeats until the concatenated text fits into a single model window. Finally, a Fine-Grained Stage produces the polished final summary.

Summ$^N$ is backbone-agnostic, meaning it can utilize various pretrained models like BART, T5, or PEGASUS as its underlying engine.

Experimental Evaluation

The researchers evaluated Summ$^N$ across several challenging long-text domains:

  • Meetings: AMI, ICSI, and QMSum (query-based), which are notoriously difficult due to speaker overlaps and disfluencies.
  • TV Scripts: SummScreen, featuring transcripts from soap operas and TV shows.
  • Government Reports: GovReport, consisting of large-scale reports from the U.S. Government Accountability Office (GAO).

Results and Key Insights

Summ$^N$ consistently outperformed previous state-of-the-art (SOTA) models across all tested datasets. For instance, on the ICSI meeting dataset, it improved ROUGE-2 and ROUGE-L scores by nearly 2 points over the previous best model, HMNet. On the SummScreen-FD dataset, it achieved a significant jump in ROUGE-1 from 25.90 to 32.48.

ROUGE-1 scores across various datasets, showing the performance gains achieved at each stage of the Summ$^N$ framework.

Key insights from the study include:

  • The Pyramid Effect: Treating summarization as recursive compression allows the model to gain a more global perspective at each stage, effectively "zooming out" on the document.
  • Preserving Transformer Integrity: By using standard attention mechanisms, Summ$^N$ leverages the high-quality representations from original pretraining, unlike "efficient" Transformers that sparsify attention.
  • Linear Complexity: Summ$^N$ reduces the quadratic time complexity ($O(n^2)$) of standard Transformers to effectively linear ($O(n)$) by processing fixed-size chunks.
  • Efficiency: For the GovReport dataset, Summ$^N$ reduced inference time to just 22.8% of the cost compared to a standard quadratic Transformer.

Human evaluations further supported these findings, with annotators rating Summ$^N$ significantly higher in readability and coverage, noting its superior ability to capture the entire narrative arc of long conversations.

Conclusion

Summ$^N$ provides a robust and efficient solution for long-form summarization by bypassing hardware constraints through a multi-stage approach. While the current framework uses separate models for each stage—increasing parameter count—future work may explore parameter sharing and more sophisticated, discourse-aware segmentation to further refine the process.

Summary updated

BibTeX
@inproceedings{zhang2022summn,
  author = {Yusen Zhang and Ansong Ni and Ziming Mao and Chen Henry Wu and Chenguang Zhu and Budhaditya Deb and Ahmed Hassan Awadallah and Dragomir R. Radev and Rui Zhang},
  title = {{SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.112},
}
Figure