Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval

John Giorgi, Luca Soldaini, Bo Wang, Gary Bader, Kyle Lo, Lucy Lu Wang, Arman Cohan

TL;DR

This paper formalizes "open-domain" multi-document summarization (MDS), a realistic setting where models must first retrieve relevant documents from a large index before summarizing them. The authors reveal that state-of-the-art summarizers are surprisingly brittle, suffering significant performance drops when faced with imperfect retrieval—particularly the introduction of irrelevant documents—though robustness can be improved through noise-aware fine-tuning.

What Problem Does This Paper Address?

Most research in multi-document summarization (MDS) operates in a "goldilocks" environment: the model is handed a pre-curated set of highly relevant documents and asked to produce a summary. In the real world, however, we usually start with an information need (a query) and a massive, messy database (like the web or a library of scientific papers).

This paper bridges the gap by investigating Open-Domain MDS. In this setting, a system must first act as a retriever to find relevant documents and then as a summarizer to synthesize them. The core challenge is that Information Retrieval (IR) is never perfect. The authors set out to quantify exactly how much these retrieval errors—like missing a key document or including a "noisy" irrelevant one—degrade the quality of the final summary.

Figure 1: “Traditional” MDS assumes a topic-related set of documents is given at train and test time. Here, we investigate the more challenging “open-domain” setting, where the document set must be retrieved given a query.

Methods

The authors employ a "retrieve-then-summarize" pipeline:

  1. Retrieval: They use two representative models: BM25 (a classic sparse, keyword-based retriever) and Contriever (a modern dense, embedding-based retriever).
  2. Summarization: They test several state-of-the-art transformer-based models, including PRIMERA, PEGASUS, LED, and LSG-BART. They also evaluate Large Language Models (LLMs) like GPT-3.5-turbo using In-Context Learning (ICL).
  3. Perturbation Analysis: To pinpoint exactly what breaks these models, the researchers systematically "attacked" the input document sets with various perturbations:
  4. Addition: Adding irrelevant documents.
  5. Deletion: Removing relevant documents.
  6. Replacement: Swapping relevant docs for irrelevant ones.
  7. Duplication: Adding copies of existing docs.
  8. Sorting: Shuffling the order of documents.
  9. Backtranslation: Paraphrasing documents to test sensitivity to minor token-level changes.
Figure 4: Graphical depiction of perturbations. Dashed line indicates input documents. Uncoloured documents are ground-truth, green have been added, red removed and blue modified. Unique documents are lettered.

Datasets

The study utilizes five diverse MDS datasets to ensure broad conclusions:

  • Multi-News & WCEP-10: News articles and event summaries.
  • Multi-XScience: Scientific literature (related work sections).
  • MS^2 & Cochrane: Medical studies and systematic reviews.

Key Insights and Ideas

  • The "Stronger" They Are, The Harder They Fall: Surprisingly, the most performant summarizers (those that do best on "clean" data) are often the most sensitive to retrieval errors. They have "more to lose" compared to weaker models that might already be relying on simple heuristics.
  • Irrelevance is the Enemy: Models are far more sensitive to Additions (irrelevant noise) than to Deletions (missing information). This suggests that in open-domain pipelines, we should tune retrievers for precision (making sure what we get is relevant) rather than just recall (trying to get everything).
  • Robustness to Order and Duplicates: In a win for practical deployment, the authors found that summarizers are generally indifferent to the order of documents and the presence of duplicates. You don't need to spend excessive compute on perfect re-ranking or deduplication for the sake of the summarizer.
  • Training for the Real World: Summarizers are typically trained on perfect "gold" document sets. The authors show that fine-tuning these models on actual retrieved (and therefore noisy) documents significantly improves their robustness in open-domain settings.
Figure 3: Fine-tuning summarizers in the open-domain. Additional training on retrieved documents can reduce sensitivity to imperfect retrieval; often at the cost of performance on ground-truth documents. Dashed grey line represents no change in performance.

Key Results

  • Quantitative Degradation: On the Multi-News dataset, the PRIMERA model saw its ROUGE-Avg F1 score drop by 7.39 points and its BERTScore by 10.33 points when moving from gold documents to retrieved ones.
  • Human Validation: Human annotators consistently preferred summaries generated from "gold" documents over those from retrieved sets, noting significant drops in both coverage (how much info is captured) and informativeness (how useful that info is).
  • LLM Performance: While GPT-3.5-turbo (via ICL) underperformed fine-tuned models overall, it showed similar trends in performance degradation, suggesting that even massive models aren't immune to retrieval noise.

Limitations and Future Work

  • Idealized Queries: The study uses reference summaries as "pseudo-queries" to bootstrap the task. In reality, user queries are often much shorter and more ambiguous, which would likely make retrieval even harder.
  • Metric Reliability: The authors acknowledge that automated metrics like ROUGE don't always perfectly align with human nuance, though their human study helped validate the main findings.
  • Future Directions: The paper calls for the creation of new, high-quality MDS datasets specifically annotated with real-world queries and large-scale document indices to move beyond "bootstrapping" and toward purpose-built open-domain systems.

Summary updated

Abstract

Multi-document summarization (MDS) assumes a set of topic-related documents are provided as input. In practice, this document set is not always available; it would need to be retrieved given an information need, i.e. a question or topic statement, a setting we dub "open-domain" MDS. We study this more challenging setting by formalizing the task and bootstrapping it using existing datasets, retrievers and summarizers. Via extensive automatic and human evaluation, we determine: (1) state-of-the-art summarizers suffer large reductions in performance when applied to open-domain MDS, (2) additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents. Based on our results, we provide practical guidelines to enable future work on open-domain MDS, e.g. how to choose the number of retrieved documents to summarize. Our results suggest that new retrieval and summarization methods and annotated resources for training and evaluation are necessary for further progress in the open-domain setting.
BibTeX
@inproceedings{giorgi2023open,
  author = {John Giorgi and Luca Soldaini and Bo Wang and Gary Bader and Kyle Lo and Lucy Lu Wang and Arman Cohan},
  title = {{Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.549},
}
Figure