TL;DR
This paper formalizes "open-domain" multi-document summarization (MDS), a realistic setting where models must first retrieve relevant documents from a large index before summarizing them. The authors reveal that state-of-the-art summarizers are surprisingly brittle, suffering significant performance drops when faced with imperfect retrieval—particularly the introduction of irrelevant documents—though robustness can be improved through noise-aware fine-tuning.
What Problem Does This Paper Address?
Most research in multi-document summarization (MDS) operates in a "goldilocks" environment: the model is handed a pre-curated set of highly relevant documents and asked to produce a summary. In the real world, however, we usually start with an information need (a query) and a massive, messy database (like the web or a library of scientific papers).
This paper bridges the gap by investigating Open-Domain MDS. In this setting, a system must first act as a retriever to find relevant documents and then as a summarizer to synthesize them. The core challenge is that Information Retrieval (IR) is never perfect. The authors set out to quantify exactly how much these retrieval errors—like missing a key document or including a "noisy" irrelevant one—degrade the quality of the final summary.
Methods
The authors employ a "retrieve-then-summarize" pipeline:
- Retrieval: They use two representative models: BM25 (a classic sparse, keyword-based retriever) and Contriever (a modern dense, embedding-based retriever).
- Summarization: They test several state-of-the-art transformer-based models, including PRIMERA, PEGASUS, LED, and LSG-BART. They also evaluate Large Language Models (LLMs) like GPT-3.5-turbo using In-Context Learning (ICL).
- Perturbation Analysis: To pinpoint exactly what breaks these models, the researchers systematically "attacked" the input document sets with various perturbations:
- Addition: Adding irrelevant documents.
- Deletion: Removing relevant documents.
- Replacement: Swapping relevant docs for irrelevant ones.
- Duplication: Adding copies of existing docs.
- Sorting: Shuffling the order of documents.
- Backtranslation: Paraphrasing documents to test sensitivity to minor token-level changes.
Datasets
The study utilizes five diverse MDS datasets to ensure broad conclusions:
- Multi-News & WCEP-10: News articles and event summaries.
- Multi-XScience: Scientific literature (related work sections).
- MS^2 & Cochrane: Medical studies and systematic reviews.
Key Insights and Ideas
- The "Stronger" They Are, The Harder They Fall: Surprisingly, the most performant summarizers (those that do best on "clean" data) are often the most sensitive to retrieval errors. They have "more to lose" compared to weaker models that might already be relying on simple heuristics.
- Irrelevance is the Enemy: Models are far more sensitive to Additions (irrelevant noise) than to Deletions (missing information). This suggests that in open-domain pipelines, we should tune retrievers for precision (making sure what we get is relevant) rather than just recall (trying to get everything).
- Robustness to Order and Duplicates: In a win for practical deployment, the authors found that summarizers are generally indifferent to the order of documents and the presence of duplicates. You don't need to spend excessive compute on perfect re-ranking or deduplication for the sake of the summarizer.
- Training for the Real World: Summarizers are typically trained on perfect "gold" document sets. The authors show that fine-tuning these models on actual retrieved (and therefore noisy) documents significantly improves their robustness in open-domain settings.
Key Results
- Quantitative Degradation: On the Multi-News dataset, the PRIMERA model saw its ROUGE-Avg F1 score drop by 7.39 points and its BERTScore by 10.33 points when moving from gold documents to retrieved ones.
- Human Validation: Human annotators consistently preferred summaries generated from "gold" documents over those from retrieved sets, noting significant drops in both coverage (how much info is captured) and informativeness (how useful that info is).
- LLM Performance: While GPT-3.5-turbo (via ICL) underperformed fine-tuned models overall, it showed similar trends in performance degradation, suggesting that even massive models aren't immune to retrieval noise.
Limitations and Future Work
- Idealized Queries: The study uses reference summaries as "pseudo-queries" to bootstrap the task. In reality, user queries are often much shorter and more ambiguous, which would likely make retrieval even harder.
- Metric Reliability: The authors acknowledge that automated metrics like ROUGE don't always perfectly align with human nuance, though their human study helped validate the main findings.
- Future Directions: The paper calls for the creation of new, high-quality MDS datasets specifically annotated with real-world queries and large-scale document indices to move beyond "bootstrapping" and toward purpose-built open-domain systems.
Summary updated