Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model

Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, Dragomir R. Radev

Abstract

Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, Dragomir Radev. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

2019.

Introduction

Multi-document summarization (MDS) is a critical task in natural language processing that involves synthesizing information from multiple source documents into a single, coherent summary. While single-document summarization (SDS) has flourished due to large-scale datasets like CNN/Daily Mail, MDS research has historically been limited by small-scale benchmarks such as DUC 2004 and TAC 2011, which contain only a few hundred examples. This "data bottleneck" has hindered the development of modern deep learning models for MDS.

In this paper, the authors introduce Multi-News, the first large-scale open-source dataset for MDS, and Hi-MAP, a hierarchical model designed to address the unique challenges of multi-source synthesis, such as redundancy and conflicting information.

The Multi-News Dataset

The Multi-News dataset consists of 56,216 article-summary pairs. The data was sourced from newser.com, where professional editors write summaries of news stories based on multiple linked source articles (ranging from 2 to 10 sources per summary).

Comparison of extractive fragment coverage and density across different datasets.

As shown in the figure above, Multi-News offers a distinct distribution of extractiveness compared to traditional datasets. While it maintains high coverage, its lower fragment density suggests that the summaries are more abstractive and less prone to simple "copy-pasting" from the source, making it a more challenging and realistic benchmark for neural models.

The Hi-MAP Model

To leverage this large-scale data, the authors propose the Hierarchical MMR-Attention Pointer-generator (Hi-MAP). The model addresses two key MDS requirements: understanding document structure and managing redundancy.

Architecture of the Hi-MAP model featuring hierarchical representations and an integrated MMR module.

The architecture incorporates two primary innovations:

  1. Hierarchical Sentence Representation: A word-level Bi-LSTM processes individual words, followed by a sentence-level LSTM that captures the relationships between sentences across different sources. This allows the model to "understand" the document structure rather than treating the input as a single long string.
  2. Integrated MMR-Attention: The model "neuralizes" the classic Maximal Marginal Relevance (MMR) algorithm. By integrating MMR directly into the pointer-generator's attention mechanism, the model learns to prioritize information that is relevant to the context but not yet covered in the generated summary, effectively reducing redundancy during the generation process.

Key Results and Insights

The authors' experiments demonstrate that models trained on Multi-News generalize well to other MDS tasks:

  • Quantitative Performance: Hi-MAP achieved a ROUGE-1 score of 43.47 on the Multi-News test set, outperforming standard pointer-generator baselines.
  • Non-Redundancy: In head-to-head human comparisons, Hi-MAP was chosen as the "best" system significantly more often than other neural models specifically for its ability to avoid repetitive "looping" behavior.
  • Zero-Shot Transfer: Models trained on Multi-News showed strong performance on the much smaller DUC 2004 dataset without further fine-tuning, proving the dataset's value as a robust training ground for the MDS task.

Conclusion

Multi-News fills a significant gap in the summarization landscape by providing the first large-scale resource for multi-document tasks. By combining this data with the Hi-MAP model, the authors demonstrate that explicitly modeling hierarchy and redundancy is essential for moving beyond simple concatenation in MDS. Future work may address current limitations, such as input truncation due to memory constraints and more complex inter-document modeling.

Summary updated

BibTeX
@inproceedings{fabbri2019multi,
  author = {Alexander R. Fabbri and Irene Li and Tianwei She and Suyi Li and Dragomir R. Radev},
  title = {{Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=MC12hXkq-w-},
}
Figure