Multi-document summarization (MDS) is significantly more complex than summarizing a single text. A model must not only identify salient information but also cross-reference facts, resolve contradictions, and eliminate redundancies across a cluster of related documents. Despite this complexity, MDS research has long been throttled by a "data bottleneck": while single-document datasets (SDS) like CNN/DailyMail offer hundreds of thousands of examples, traditional MDS benchmarks often contain fewer than 100 human-written summaries.
The high cost of human annotation—requiring experts to digest dozens of articles to produce one summary—has forced researchers to develop creative strategies to train high-capacity neural models without massive "gold-standard" datasets.
Leveraging Single-Document Knowledge
The most immediate solution to MDS data scarcity is to "borrow" the knowledge captured by models trained on abundant single-document data. Because SDS and MDS share fundamental linguistic requirements—like sentence compression and abstractive rewriting—researchers have focused on adapting these pre-trained architectures.
Jianmin Zhang, Xiaojun Wan, and colleagues demonstrated in Towards a Neural Network Approach to Abstractive Multi-Document Summarization that a hierarchical "docset encoder" can effectively transfer knowledge from SDS tasks. By using a concentrated attention mechanism, the model learns to focus on the most relevant parts of a document set, effectively treating the multi-document problem as an extension of the single-document one. Similarly, Hanqi Jin and Xiaojun Wan proposed joint learning frameworks that train on both tasks simultaneously, using a shared Transformer encoder to ensure the model benefits from the scale of SDS while specializing for the aggregation needs of MDS.
Using External Saliency Filters
If humans are too expensive to hire for annotation, researchers look for "free" signals of importance elsewhere in the digital ecosystem. This approach treats external data as a proxy for human judgment.
In TGSum, Ziqiang Cao, Wenjie Li, and Furu Wei used Twitter as a crowd-sourced "saliency filter." By matching news articles with related tweets, they used the collective attention of social media users to identify which facts were most important, automatically synthesizing reference summaries via Integer Linear Programming.
Another strategy involves using text classification data to guide the summarization style. The TCSum framework by Wenjie Li and Ziqiang Cao leverages category-specific transformation matrices. By learning from abundant classification data, the model "knows" that a summary of a natural disaster should prioritize dates and locations, while a biography should focus on career milestones, even without seeing many explicit MDS examples for those categories.
Synthetic Data and Self-Training
When external proxies are unavailable, researchers turn to self-training and synthetic generation to create "silver-standard" data. The intuition here is that while generating a perfect summary is hard, identifying consensus across documents is a task that can be partially automated.
Annie Louis and Joshua Maynez introduced OpineSum, which uses textual entailment to find "consensus" opinions across thousands of user reviews. By identifying statements that are supported by multiple sources, the system can automatically generate high-quality training data for opinion summarization—a domain where human annotation is particularly taxing.
For modern Large Language Models (LLMs), Gabrielle Kaili-May Liu and Avi Caciularu developed MDCure, a pipeline that generates synthetic multi-document instruction-following data. This allows mid-sized models to learn complex cross-document reasoning without requiring massive human-labeled corpora.
Unsupervised and Few-Shot Learning
A final direction involves moving away from the need for large-scale supervision entirely. If the model can understand the "representative" nature of a summary, it can perform MDS in an unsupervised or few-shot manner.
Arthur Brazinskas, Mirella Lapata, and Ivan Titov demonstrated in FEWSUM that a few dozen examples are often sufficient to bridge the gap if the model is properly initialized. In the unsupervised realm, Haopeng Zhang and Jiawei Zhang proposed the Subset Representative Index (SRI), a metric that treats summarization as a holistic selection problem. Instead of scoring sentences individually, the model selects a collection of sentences that collectively maximize information and minimize redundancy, bypassing the need for reference summaries during training.
The Path to Large-Scale Benchmarks
While these workarounds have sustained the field, the ultimate goal remains the creation of massive, authentic MDS datasets. Recent efforts like the Wikipedia Current Events Portal (WCEP) dataset have begun to provide the scale necessary for high-capacity models, offering over 10,000 event clusters. By pairing professional news blurbs with thousands of related articles, these benchmarks finally allow MDS models to move from "data-starved" adaptation to "data-rich" direct training.