Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters

Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, Markus DreyerNAACL 2021 · OpenReview

Abstract

Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, Markus Dreyer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2021.

Introduction

Multi-Document Summarization (MDS) represents a significant challenge in natural language processing, requiring the synthesis of information from multiple sources into a single, coherent narrative. Researchers from UNC Chapel Hill and Amazon Alexa have identified two primary hurdles in this domain: the quadratic memory bottleneck of standard Transformer models and the difficulty of capturing inter-document context. To address these, they propose BART-Long-Graph, a dual-encoder system that integrates long-form text processing with semantic graph representations.

Methodology

The BART-Long-Graph architecture introduces three significant innovations to the MDS pipeline:

1. BART-Long with Linear Attention

Standard Transformers suffer from $O(n^2)$ complexity due to full self-attention. The authors replace this with Longformer-style attention, utilizing sliding windows (local attention) and global tokens. This shift reduces complexity to $O(n)$, allowing the model to process sequences up to 2,000 tokens, far exceeding the typical 500-token limit.

2. Semantic Graph Construction

To capture relationships across documents, the system employs Open Information Extraction (OIE) to extract (Subject-Predicate-Object) triplets. These triplets are merged into a unified semantic graph that represents the core factual structure of the document cluster.

Figure 3: Illustration of input documents, the resulting semantic graph, and its linearized text representation.

3. Dual-Encoder Fusion

The model utilizes two distinct encoders: a Text Encoder for raw linguistic data and a Graph Encoder for the linearized semantic graph. These streams are fused early in the encoding process, enabling the model to combine structural knowledge with textual nuances.

Experimental Results

The model was evaluated on the Multi-News dataset and tested for transferability on DUC-2004.

Quantitative Performance

BART-Long-Graph achieved state-of-the-art (SOTA) results on the Multi-News dataset. As shown in the table below, the model consistently outperformed previous benchmarks across all ROUGE metrics.

Figure 6: Performance comparison of BART-Long-Graph against previous state-of-the-art models on the Multi-News dataset.

Qualitative Insights

  • Enhanced Abstraction: The inclusion of graph encodings encourages the model to be more abstractive. Rather than simple extraction, the model uses the graph as a factual "skeleton" to generate original phrasing.
  • Factual Consistency: Human evaluations indicated that graph-enhanced summaries were significantly more informative (25.5% vs 17.0%) and more factually consistent (72% vs 68%) than text-only baselines.
  • Generalization: The model demonstrated strong zero-shot performance on the DUC-2004 dataset, suggesting that semantic graphs provide a robust, generalizable representation of information.

Conclusion and Future Work

BART-Long-Graph effectively mitigates the computational and contextual challenges of MDS. While the model sets a new benchmark, the authors note that "hallucinations"—such as misattributing sources—remain a challenge. Future research will explore more sophisticated graph construction and training objectives designed to further penalize factual inconsistencies.

Summary updated

BibTeX
@inproceedings{pasunuru2021efficiently,
  author = {Ramakanth Pasunuru and Mengwen Liu and Mohit Bansal and Sujith Ravi and Markus Dreyer},
  title = {{Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=vKvLvMA8zCj},
}
Figure