Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval

John Michael Giorgi, Luca Soldaini, Bo Wang, Gary D. Bader, Kyle Lo, Lucy Lu Wang, Arman Cohan

TL;DR

This paper formalizes "Open-Domain Multi-document Summarization" (MDS), a realistic setting where a model must retrieve its own documents from a massive index before summarizing them. The researchers discover that while state-of-the-art summarizers are highly brittle when faced with imperfect retrieval, they can be made significantly more robust through targeted training on noisy, retrieved data.

What Problem Does This Paper Address?

In the "traditional" multi-document summarization (MDS) research setting, models are handed a neat, pre-selected stack of relevant documents and told to summarize them. In the real world, this is rarely the case. Usually, you start with an "information need"—a question or a topic—and you have to find the relevant documents yourself from a massive, messy collection (like the web or a corporate database).

The authors dub this more realistic challenge Open-Domain MDS. The core problem is that information retrieval (IR) is never perfect. A retriever might miss key documents, include irrelevant ones, or return duplicates. This paper investigates a critical gap: we don't actually know how sensitive our best summarization models are to these inevitable retrieval errors.

Methods

The researchers established a "retrieve-then-summarize" framework to study this task:

  1. Formalization & Bootstrapping: Since no large-scale "Open-Domain MDS" datasets exist, they adapted five existing MDS datasets. They used the human-written reference summaries as "pseudo-queries" to search against an index containing all documents from the original dataset.
  2. The Pipeline: They paired two types of retrievers—BM25 (a classic keyword-based sparse retriever) and Contriever (a modern neural dense retriever)—with several state-of-the-art transformer-based summarizers (like PRIMERA, PEGASUS, and LED).
  3. Perturbation Analysis: To pinpoint exactly what "breaks" a model, they systematically injected synthetic errors into the input:
  4. Addition/Replacement: Adding irrelevant documents.
  5. Deletion: Removing relevant documents.
  6. Duplication: Repeating documents.
  7. Sorting: Shuffling the document order.
  8. Backtranslation: Making subtle token-level changes to simulate paraphrasing noise.
  9. Robustness Training: They tested whether fine-tuning summarizers on documents actually returned by a retriever (rather than "gold" documents) would help them handle noise better.

Datasets

The study utilized a diverse range of benchmarks:

  • News: Multi-News and WCEP-10.
  • Scientific Literature: Multi-XScience (summarizing related work).
  • Medical/Systematic Reviews: MS^2 and Cochrane.

Key Insights and Ideas

  • The "Stronger is Brittler" Paradox: Surprisingly, the most powerful summarization models (those with the highest scores on clean data) suffered the most dramatic performance collapses when retrieval was imperfect. They have "more to lose."
  • Irrelevance is the Enemy: Models are incredibly sensitive to "distractor" documents. Adding even a few irrelevant documents to the input caused sharp drops in summary quality.
  • Robustness to Messiness: On the flip side, models were remarkably unfazed by duplicate documents or changes in document order. This suggests that the internal attention mechanisms of these transformers are already quite good at deduplication and handling unordered sets.
  • Training on Noise Works: The "brittleness" isn't permanent. Models trained on retrieved (and therefore imperfect) document sets became much more resilient to retrieval errors at test time, often without sacrificing much performance on "clean" data.
  • Precision > Recall for Summarization: The results suggest that for open-domain pipelines, it is better to tune the retriever for precision (fewer, highly relevant docs) rather than recall (getting everything, but including junk).

Key Results

  • Performance Drop: When moving from "gold" documents to retrieved documents, models saw significant drops in ROUGE-Avg F1 scores (e.g., a 7.39 point drop for PRIMERA on Multi-News).
  • Human Preference: In a head-to-head evaluation, human annotators overwhelmingly preferred summaries generated from "gold" documents over those from retrieved documents, citing better coverage and informativeness.
  • Zero-Shot LLMs: While GPT-3.5 (via in-context learning) was more robust to noise than some smaller fine-tuned models, it still underperformed compared to specialized models like PRIMERA that were fine-tuned on the specific task.

Limitations and Future Work

  • Pseudo-Queries: Using a full reference summary as a query is an "idealized" scenario. Real-world user queries are typically much shorter and more ambiguous, which would likely make retrieval even harder.
  • Evaluation Metrics: The authors acknowledge that metrics like ROUGE and BERTScore don't always capture factual consistency or "hallucinations" caused by irrelevant retrieved text.
  • Future Directions: The paper calls for the creation of "true" open-domain MDS datasets with human-annotated queries. They also suggest exploring "span-level" retrieval—finding specific relevant passages across many documents rather than retrieving entire files.

Summary updated

Abstract

Multi-document summarization (MDS) assumes a set of topic-related documents are provided as input. In practice, this document set is not always available; it would need to be retrieved given an information need, i.e. a question or topic statement, a setting we dub "open-domain" MDS. We study this more challenging setting by formalizing the task and bootstrapping it using existing datasets, retrievers and summarizers. Via extensive automatic and human evaluation, we determine: (1) state-of-the-art summarizers suffer large reductions in performance when applied to open-domain MDS, (2) additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents. Based on our results, we provide practical guidelines to enable future work on open-domain MDS, e.g. how to choose the number of retrieved documents to summarize. Our results suggest that new retrieval and summarization methods and annotated resources for training and evaluation are necessary for further progress in the open-domain setting.
BibTeX
@inproceedings{giorgi2023open,
  author = {John Michael Giorgi and Luca Soldaini and BO WANG and Gary D. Bader and Kyle Lo and Lucy Lu Wang and Arman Cohan},
  title = {{Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.549},
}
Figure