Abstract
Automatic generation of summaries from multiple news articles is a valuable tool as the number of online publications grows rapidly. Single document summarization (SDS) systems have benefited from advances in neural encoder-decoder model thanks to the availability of large datasets. However, multi-document summarization (MDS) of news articles has been limited to datasets of a couple of hundred examples. In this paper, we introduce Multi-News, the first large-scale MDS news dataset. Additionally, we propose an end-to-end model which incorporates a traditional extractive summarization model with a standard SDS model and achieves competitive results on MDS datasets. We benchmark several methods on Multi-News and release our data and code in hope that this work will promote advances in summarization in the multi-document setting.
Introduction
While single-document summarization (SDS) has flourished thanks to massive datasets like CNN/Daily Mail, multi-document summarization (MDS) has historically been limited by the lack of large-scale training data. Existing MDS benchmarks often contain only a few hundred examples, making it nearly impossible to train modern deep-learning models from scratch. MDS is inherently more difficult than SDS because a model must not only identify salient information but also synthesize it from multiple, often conflicting or redundant, sources.
To address this, researchers from Yale University have released Multi-News, the first large-scale dataset for MDS in the news domain, featuring over 56,000 article-summary pairs. Alongside the dataset, they introduced Hi-MAP, a hierarchical model designed to handle the complexities of multi-source information.
Methods: The Hi-MAP Architecture
The authors propose a novel architecture called Hi-MAP (Hierarchical MMR-Attention Pointer-generator). It builds upon the standard pointer-generator network with two major innovations:
- Hierarchical Encoding: Instead of treating multiple documents as one long flat sequence, the model uses a word-level Bi-LSTM to create token representations, which are then aggregated into sentence-level representations. This preserves the structural boundaries of the input documents.
- End-to-End MMR Attention: Maximal Marginal Relevance (MMR) is a classic extractive technique used to balance relevance with novelty. Hi-MAP incorporates a learned version of MMR directly into the attention mechanism. The model calculates an MMR score for each sentence at every decoding step and uses it to weight the word-level attention, effectively guiding the abstractive decoder toward salient, non-redundant content.
Dataset Analysis
The Multi-News dataset consists of 56,216 article-summary pairs. The summaries are professionally written by editors at newser.com, who synthesize information from multiple cited news sources. On average, each example includes ~2.3 source articles and a summary of ~260 words.
Analysis shows that Multi-News is more abstractive than older MDS datasets like DUC or TAC. In these older datasets, summaries often rely heavily on "cutting and pasting" sentences, whereas Multi-News summaries exhibit higher levels of rephrasing and synthesis. This is reflected in the distribution of extractive fragment coverage and density across the datasets.
Key Results
- Performance: On the Multi-News test set, Hi-MAP achieved a ROUGE-1 score of 43.47, outperforming standard pointer-generator baselines.
- Redundancy Reduction: In human evaluations, Hi-MAP was chosen as the "best" for non-redundancy significantly more often than other neural models, validating the effectiveness of the integrated MMR module.
- Transferability: When trained on Multi-News and tested on the smaller DUC 2004 dataset, the models showed strong zero-shot performance, proving that the patterns learned from Multi-News generalize well to other news summarization tasks.
Discussion and Future Work
Despite its success, the model faces hurdles such as input truncation due to memory constraints, which limits the amount of source text the model can process. Future work may explore more complex graph-based interactions between documents and the integration of the MMR-attention mechanism into Transformer-based architectures to better handle long-range dependencies.
Summary updated