Towards Multi-Document Question Answering in Scientific Literature: Pipeline, Dataset, and Evaluation

Hui Huang, Julien Velcin, Yacine KessaciEMNLP 2025

TL;DR

Answering complex questions from scientific literature often requires synthesizing information from multiple papers. This paper introduces MDA-QA, a new dataset and pipeline for creating such multi-document question-answer pairs, demonstrating that current retrieval methods struggle to identify all necessary documents, highlighting a significant gap in multi-document reasoning.

What Problem Does This Paper Address?

Researchers frequently need to answer questions that draw upon knowledge scattered across multiple scientific articles. While existing Question Answering (QA) systems are adept at handling single documents or simple multi-hop reasoning, they often fall short when faced with questions demanding a deep synthesis of information from several distinct sources. Existing multi-document QA datasets either don't strictly enforce this cross-document synthesis or fail to leverage the inherent structure of scientific literature, such as citation networks, to guide the process. This gap makes it difficult to accurately assess and improve systems designed for complex scientific inquiry.

Methods

The authors propose a three-module pipeline for generating multi-document QA pairs:

  1. Community Building: They leverage citation networks from academic papers. Using the Speaker-Listener Label Propagation (SLPA) algorithm, they identify "thematically coherent communities" of papers. This approach is favored over others like Louvain because it produces smaller, more manageable communities, which are better suited for LLM processing.
  2. QA Generation: Once communities are formed, a Large Language Model (LLM) is prompted to generate single-sentence questions and concise, multi-document answers. The prompt specifically instructs the LLM to cross-reference at least two documents and to systematically examine overlaps. An automatic cross-check is integrated to ensure the generated QA truly requires more than one document.
  3. Automated Quality Assurance: To ensure that generated QA pairs genuinely require multi-document synthesis, an automated filtering process is employed. For each QA pair, the LLM attempts to answer the question using individual supporting documents. If an answer derived from a single document is found to be as good as or better than the proposed multi-document answer, the QA pair is discarded. This process is further validated by manual expert review.

Datasets

  • SPIQA Dataset: The authors use the SPIQA dataset, which contains 25,859 peer-reviewed machine learning conference papers, as the foundation for their work.
  • MDA-QA Dataset (New Benchmark): The paper introduces the MDA-QA dataset, comprising 6,804 high-quality multi-document QA pairs. This dataset is specifically designed to require synthesizing information from multiple sources, distinguishing it from existing datasets that may only incidentally involve multiple documents.

Key Insights and Ideas

  • Citation Networks as a Structure for QA: The paper highlights the utility of citation networks not just for understanding paper relationships but as a structural backbone for identifying relevant document groups for multi-document QA.
  • LLMs as Powerful QA Generators, but Need Guidance: LLMs are capable of generating complex, multi-document QA, but their output requires careful prompting and rigorous filtering to ensure the questions are non-trivial and truly necessitate cross-document synthesis.
  • Current Retrieval Methods Fall Short: Even advanced semantic embedding models struggle to retrieve all necessary documents for multi-document QA, indicating a significant challenge in this area.
  • Document Similarity's Nuanced Impact: While more similar documents within a community can slightly improve retrieval, high similarity doesn't eliminate the fundamental difficulty of multi-document integration and inference.

Key Results

  • MDA-QA Dataset: The authors successfully generated 6,804 multi-document QA pairs, with an average of 2-3 supporting documents per question.
  • Retrieval Performance:
    • Lexical methods like BM25 perform poorly compared to neural embedding methods.
  • Semantic embedding methods (BGE, MPNET) show higher recall.
  • Augmenting document embeddings with citation neighbor information (BGE-neighbor) yielded the highest recall (around 0.55 for Recall@10 in document-level retrieval).
  • Despite improvements, recall values remain relatively low (below 0.70 even at k=50), underscoring the difficulty.
  • RAG Baseline: Using BGE-neighbor for retrieval and GPT-40 for generation improved metrics like METEOR and ROUGE-L compared to a zero-shot GPT-40 approach, suggesting that structured retrieval aids generation.
  • Expert Validation: Manual review of sampled QA pairs confirmed that the automated filtering effectively retained questions requiring multi-document synthesis and discarded those answerable by a single document.

Limitations and Future Work

  • Domain Specificity: The current MDA-QA dataset is limited to the machine learning domain. Expanding to diverse scientific fields would create more comprehensive benchmarks.
  • LLM Reliance: The QA generation and filtering processes heavily rely on LLMs, which can potentially produce hallucinations or overlook nuances. While expert review was conducted, large-scale validation is ongoing.
  • Computational Cost: The pipeline involves extensive text parsing and repeated LLM calls, making it resource-intensive and potentially limiting scalability.
  • Future Work: The authors suggest exploring graph-based methods, contextual reasoning, and knowledge integration for improved document merging and retrieval. They also propose developing more advanced multi-document reasoning models for QA. Future work may also consider hierarchical or sub-community approaches to handle larger document groups.

Summary updated

BibTeX
@inproceedings{huang2025towards,
  author = {Hui Huang and Julien Velcin and Yacine Kessaci},
  title = {{Towards Multi-Document Question Answering in Scientific Literature: Pipeline, Dataset, and Evaluation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure