Multi-Document Summarization with Centroid-Based Pretraining

Ratish Surendran Puduppully, Parag Jain, Nancy Chen, Mark SteedmanACL 2023 · DOI

Abstract

Multi-document summarization (MDS) requires models to synthesize information from multiple sources while maintaining coherence and avoiding redundancy. Current state-of-the-art models often rely on synthetic pretraining using heuristically assembled summaries, which can lead to incoherent training signals. We present Centrum, a model that utilizes the "centroid" document of a cluster—the most representative natural document—as a proxy for the summary during pretraining. This approach preserves natural coherence and entity grounding. Centrum demonstrates superior performance in zero-shot and few-shot settings across multiple benchmarks, producing more informative and coherent summaries than previous heuristic-based methods.

Introduction

The primary challenge in Multi-Document Summarization (MDS) lies in navigating the complexities of redundancy and conflicting information across multiple source documents. While single-document summarization has seen significant progress, MDS remains difficult due to the long-range dependencies and the need for cross-document synthesis.

Recent advancements, such as the PRIMERA model, have utilized "synthetic pretraining" to overcome the lack of large-scale human-annotated MDS datasets. These models create synthetic summaries by stitching together sentences from various documents in a cluster. However, this "Frankensteining" process often results in incoherent summaries with broken entity references and high redundancy. This noisy training data limits the model's ability to generalize, particularly in low-resource (zero-shot and few-shot) scenarios.

Methodology: The Centroid-Based Approach

The Centrum model addresses the limitations of synthetic pretraining by prioritizing natural coherence. Instead of assembling a summary from fragments, Centrum identifies the most representative document within a cluster to serve as a proxy summary.

1. Centroid Selection

For a given cluster of related news articles, Centrum calculates the ROUGE scores between all pairs of documents. The document that shares the most content with all others—the "centroid"—is selected as the ground-truth summary. The remaining documents in the cluster serve as the input to the model.

2. Architecture and Attention

Centrum utilizes the Longformer Encoder-Decoder (LED) architecture, which is optimized for processing the large token counts typical of MDS tasks. To effectively manage multiple source documents, the model employs special <doc-sep> tokens. Global attention is applied to these tokens, allowing the model to capture the structural relationships between different sources.

Experimental Setup

The model was pretrained on the NewSHead corpus, which contains over 369,000 news clusters. After filtering for clusters with at least three documents and a minimum summary length of 250 tokens, approximately 172,000 clusters were used for training.

Evaluation was conducted on three major MDS benchmarks:

  • Multi-News: A large-scale dataset of news articles with human-written summaries.
  • WCEP: Summaries derived from the Wikipedia Current Events Portal.
  • DUC 2007: A classic, smaller-scale benchmark for rigorous evaluation.

Results

Centrum consistently outperforms or matches existing state-of-the-art models, particularly in data-constrained environments.

Quantitative Performance

As shown in the performance table below, Centrum achieves significant gains in zero-shot and few-shot performance. On the Multi-News dataset, Centrum reached a ROUGE-1 (R1) score of 43.5 in a zero-shot setting, outperforming PRIMERA's 41.6.

Performance comparison of Centrum against Pegasus and PRIMERA across various datasets and shot settings.

Human Evaluation

To assess the qualitative aspects of the generated summaries, professional linguists evaluated the models based on informativeness, coherence, and non-repetition.

Human evaluation results comparing Centrum and PRIMERA on informativeness (Inform), coherence (Coh), and repetition (Rep).

Centrum was rated significantly higher in informativeness compared to PRIMERA. While it performed comparably in coherence, the use of a natural document as a pretraining target ensures that the model learns proper entity grounding and logical flow, which are often missing in heuristically pretrained models.

Discussion and Limitations

The success of Centrum suggests that "coherence is inherent, not assembled." By learning from real, human-written documents, the model avoids the pitfalls of noisy synthetic data. Furthermore, the centroid selection process acts as a natural noise filter, ignoring outlier documents that may be incorrectly clustered.

However, there are trade-offs. Centrum tends to produce longer summaries, likely a reflection of its pretraining on full documents. Additionally, the requirement for clusters with at least three documents led to the exclusion of over half of the available pretraining data. Future work may explore hybrid objectives, such as combining centroid-based pretraining with "Gap Sentence Generation" to improve data efficiency.

Conclusion

Centrum sets a new standard for multi-document summarization by leveraging the natural coherence of centroid documents. Its superior performance in zero-shot and few-shot settings demonstrates the value of high-quality, natural pretraining targets over heuristically generated synthetic data.

Reference: Puduppully, R., Jain, P., Chen, N. F., & Steedman, M. (2023). Multi-Document Summarization with Centroid-Based Pretraining. Proceedings of ACL 2023.

Summary updated

BibTeX
@inproceedings{puduppully2023multi,
  author = {Ratish Surendran Puduppully and Parag Jain and Nancy Chen and Mark Steedman},
  title = {{Multi-Document Summarization with Centroid-Based Pretraining}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.13},
}
Figure