Mitigating False-Negative Contexts in Multi-document QuestionAnswering with Retrieval Marginalization

Ansong Ni, Matt Gardner, Pradeep Dasigi

Abstract

Standard multi-document Question Answering (QA) models are often penalized during training for identifying valid evidence that was not explicitly labeled by human annotators. This paper introduces a retrieval marginalization framework that treats multiple potential evidence sets as valid by summing over the top-$m$ retrieved context candidates. By allowing the model to learn from "alternative" evidence, this approach significantly improves performance on complex reasoning tasks, achieving a new state-of-the-art on the IIRC benchmark (+5.5 F1) and substantial gains on HotpotQA (+4.1 F1).

Introduction

In multi-document QA, systems must retrieve relevant snippets from large corpora like Wikipedia and reason across them to derive an answer. Traditionally, these models are trained using "gold" evidence—specific sentences highlighted by human annotators. However, large knowledge bases are inherently redundant; the same answer can often be found through multiple different snippets or combinations of documents.

When a model retrieves a perfectly valid snippet that was not labeled as gold, standard training regimes treat it as a false negative, punishing the model for being correct via an "unauthorized" path. This creates a noisy training signal that hinders performance, especially for numerical reasoning, boolean questions, and unanswerable queries.

The Challenge of Alternative Evidence

A manual analysis of the IIRC dataset reveals that over half of the questions have at least one valid evidence snippet not labeled as "gold." In many cases, there are actually more alternative snippets available than human-labeled ones.

Examples of valid evidence snippets in IIRC and HotpotQA that are not labeled as 'gold' evidence, leading to false-negative penalties during standard training.

As shown in the figure above, a model might find the correct answer (e.g., "115") using a snippet that explicitly states the number of participants, even if the annotator pointed to a different sentence requiring subtraction. Under standard supervision, the model is penalized for selecting the more direct, unlabeled snippet.

The Retrieval Marginalization Framework

To address this, the authors propose a joint framework for retrieval and reasoning that utilizes Retrieval Marginalization. Instead of forcing the model to converge on a single gold context, the system considers a set of likely context candidates.

The proposed retrieval marginalization framework, which decomposes retrieval into document and evidence selection before marginalizing over context candidates.

Key Components:

  1. Decomposed Retrieval: The process is split into Document Selection and Evidence Retrieval. To handle irrelevant data, the retriever can select a special NULL token to effectively reject a document.
  2. Marginalization over Sets: During training, the model calculates the probability of an answer by marginalizing over the top-$m$ retrieved contexts. If a retrieved snippet helps predict the correct answer, the model is rewarded, regardless of its gold status.
  3. Heuristic Filtering & Auxiliary Loss: To prevent the model from "cheating" by marginalizing over irrelevant text that happens to contain the answer string, heuristics (like checking for relevant numbers or entities) separate valid alternative contexts from invalid ones. An auxiliary loss then encourages the model to treat invalid contexts as "unanswerable."

Experimental Evaluation

The researchers evaluated the model on two major benchmarks: IIRC (Information-seeking Iterative Retrieval Collection) and HotpotQA (Fullwiki setting).

  • IIRC Results: The model achieved a new state-of-the-art with 50.5 F1, a 5.5-point improvement over the non-marginalized baseline. The gains were most pronounced in "Yes/No" questions and numerical reasoning, where lexical overlap between the question and evidence is often low.
  • HotpotQA Results: In the challenging full-wiki setting, the model showed a 4.1 F1 improvement. Qualitative analysis confirmed the model could successfully find evidence in documents different from those used by annotators (e.g., finding biographical details in a contemporary's article rather than the subject's own page).

Conclusion

By moving away from rigid gold-label supervision and embracing the redundancy of large corpora through marginalization, this framework mitigates the harmful effects of false-negative training signals. While joint training remains computationally intensive, the results demonstrate that allowing a model to find its own path to the truth leads to more robust and accurate multi-document reasoning.

Summary updated

Abstract

Question Answering (QA) tasks requiring information from multiple documents often rely on a retrieval model to identify relevant information for reasoning. The retrieval model is typically trained to maximize the likelihood of the labeled supporting evidence. However, when retrieving from large text corpora such as Wikipedia, the correct answer can often be obtained from multiple evidence candidates. Moreover, not all such candidates are labeled as positive during annotation, rendering the training signal weak and noisy. This problem is exacerbated when the questions are unanswerable or when the answers are Boolean, since the model cannot rely on lexical overlap to make a connection between the answer and supporting evidence. We develop a new parameterization of set-valued retrieval that handles unanswerable queries, and we show that marginalizing over this set during training allows a model to mitigate false negatives in supporting evidence annotations. We test our method on two multi-document QA datasets, IIRC and HotpotQA. On IIRC, we show that joint modeling with marginalization improves model performance by 5.5 F1 points and achieves a new state-of-the-art performance of 50.5 F1. We also show that retrieval marginalization results in 4.1 QA F1 improvement over a non-marginalized baseline on HotpotQA in the fullwiki setting.
BibTeX
@inproceedings{ni2021mitigating,
  author = {Ansong Ni and Matt Gardner and Pradeep Dasigi},
  title = {{Mitigating False-Negative Contexts in Multi-document QuestionAnswering with Retrieval Marginalization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.497},
}
Figure