Abstract
Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, Markus Dreyer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
2021.
Introduction
Multi-Document Summarization (MDS) represents a significant challenge in natural language processing, requiring the synthesis of information from multiple sources into a single, coherent narrative. Researchers from UNC Chapel Hill and Amazon Alexa have identified two primary hurdles in this domain: the quadratic memory bottleneck of standard Transformer models and the difficulty of capturing inter-document context. To address these, they propose BART-Long-Graph, a dual-encoder system that integrates long-form text processing with semantic graph representations.
Methodology
The BART-Long-Graph architecture introduces three significant innovations to the MDS pipeline:
1. BART-Long with Linear Attention
Standard Transformers suffer from $O(n^2)$ complexity due to full self-attention. The authors replace this with Longformer-style attention, utilizing sliding windows (local attention) and global tokens. This shift reduces complexity to $O(n)$, allowing the model to process sequences up to 2,000 tokens, far exceeding the typical 500-token limit.
2. Semantic Graph Construction
To capture relationships across documents, the system employs Open Information Extraction (OIE) to extract (Subject-Predicate-Object) triplets. These triplets are merged into a unified semantic graph that represents the core factual structure of the document cluster.
3. Dual-Encoder Fusion
The model utilizes two distinct encoders: a Text Encoder for raw linguistic data and a Graph Encoder for the linearized semantic graph. These streams are fused early in the encoding process, enabling the model to combine structural knowledge with textual nuances.
Experimental Results
The model was evaluated on the Multi-News dataset and tested for transferability on DUC-2004.
Quantitative Performance
BART-Long-Graph achieved state-of-the-art (SOTA) results on the Multi-News dataset. As shown in the table below, the model consistently outperformed previous benchmarks across all ROUGE metrics.
Qualitative Insights
- Enhanced Abstraction: The inclusion of graph encodings encourages the model to be more abstractive. Rather than simple extraction, the model uses the graph as a factual "skeleton" to generate original phrasing.
- Factual Consistency: Human evaluations indicated that graph-enhanced summaries were significantly more informative (25.5% vs 17.0%) and more factually consistent (72% vs 68%) than text-only baselines.
- Generalization: The model demonstrated strong zero-shot performance on the DUC-2004 dataset, suggesting that semantic graphs provide a robust, generalizable representation of information.
Conclusion and Future Work
BART-Long-Graph effectively mitigates the computational and contextual challenges of MDS. While the model sets a new benchmark, the authors note that "hallucinations"—such as misattributing sources—remain a challenge. Future research will explore more sophisticated graph construction and training objectives designed to further penalize factual inconsistencies.
Summary updated