Abstract
Multi-document summarization (MDS) typically focuses on homogeneous document sets, such as multiple reports on a single event. However, real-world applications often require summarizing heterogeneous collections—large groups of documents covering diverse stories. This paper introduces LSMD (Large-Scale Multi-Document Summarization), a three-stage unsupervised framework that combines extractive techniques with GPT-2-enhanced sentence fusion and a specialized Reverse Coreference Resolution (RCR) module. By leveraging BERT embeddings for initial extraction and a Pointer Generator Network for readability, LSMD achieves significant performance gains over state-of-the-art unsupervised models, particularly in large-scale settings.
Introduction
The challenge of summarizing large collections of documents often lies in their "heterogeneity." While traditional MDS methods excel at finding a common thread among articles describing the same event, they often fail when faced with diverse datasets like a week's worth of financial news or thousands of research papers. In these cases, traditional methods either produce redundant summaries or lose focus by attempting to find a non-existent central topic. LSMD addresses this by functioning as an intelligent filter that scales effectively without the need for expensive labeled training data.
Methodology
The LSMD framework follows a three-stage pipeline designed to extract, de-duplicate, and refine information:
- Per-Document Information Extraction: The system identifies key sentences using a modified TextRank algorithm. By incorporating BERT embeddings to calculate sentence similarity, the model recognizes semantically related ideas even when they lack exact word overlap.
- Information Redundancy Minimization: Extracted sentences are grouped via hierarchical clustering. To merge these into concise summaries, a word-graph approach is used for sentence fusion. Crucially, GPT-2 is integrated into the fusion process to score potential paths, ensuring the resulting sentences are grammatically sound and readable.
- Reverse Coreference Resolution (RCR): To improve flow, a Pointer Generator Network intelligently re-inserts pronouns (e.g., replacing "Elon Musk" with "he" where appropriate), reversing the typical preprocessing step that replaces pronouns with proper nouns to avoid ambiguity.
Datasets
To evaluate performance on heterogeneous data, the authors developed 12 new dataset variations based on CNN/Daily Mail and NewsRoom. These datasets were scaled to include 10 and 100 documents per group, tripling original document counts to simulate real-world redundancy levels.
Results and Key Insights
LSMD was compared against several unsupervised baselines, including TextRank and AUMDSPSF. The framework consistently outperformed all competitors in ROUGE-1 and ROUGE-2 scores across all tested datasets. Notably, the performance gap widened as the dataset scale increased; on the NewsRoom Scale 100 dataset, LSMD outperformed the second-best system by 62% in terms of ROUGE-1 scores.
Qualitative analysis and readability formulas (such as Dale-Chall) further confirmed that the GPT-2 enhancement and RCR module produced more human-like, readable summaries with lower redundancy compared to traditional graph-based methods. The use of GPT-2 as a "grammar police" for word-graph paths effectively combines the factual grounding of extractive methods with the fluency of modern language models.
Limitations and Future Work
While effective, the RCR module occasionally struggled with very long sequences, leading to repetitive word generation. Additionally, the authors identified instances of gender bias in the RCR model, where names like "Kim" were incorrectly associated with masculine pronouns due to training data imbalances. Future research will explore moving toward a unified, end-to-end model using Graph Neural Networks to further refine sentence condensation and integration.
Summary updated