Mitigating False-Negative Contexts in Multi-document Question Answering with Retrieval Marginalization

Ansong Ni, Matt Gardner, Pradeep Dasigi

Abstract

Multi-document question answering (MDQA) requires systems to retrieve relevant information from large corpora and reason across multiple snippets. Traditional training relies on "gold" evidence labels, which are often incomplete in redundant datasets like Wikipedia, leading to "false negative" penalties for valid but unlabeled retrievals. This paper introduces a joint retrieval-reasoning framework that treats context as a latent variable and employs retrieval marginalization. By summing over multiple potential evidence sets, the model learns to identify and utilize unlabeled valid information. Our approach achieves state-of-the-art performance on the IIRC and HotpotQA benchmarks, showing particular strength in handling unanswerable and binary questions.

The Problem of False-Negative Contexts

In MDQA, systems typically follow a "hard" supervision paradigm where the retriever is rewarded only for finding human-annotated "gold" snippets. However, large-scale corpora are inherently redundant; a single fact may be expressed in multiple ways across different documents. If a model retrieves a valid snippet that wasn't explicitly labeled by an annotator, it is penalized during training. This noise is particularly detrimental for complex reasoning tasks, such as numerical or Boolean (Yes/No) questions, where simple lexical overlap is insufficient.

Methodology: Retrieval Marginalization

To address the limitations of hard supervision, we propose a framework that treats the retrieved context as a latent variable. Instead of optimizing for a single labeled path, the model considers a set of potential contexts.

The proposed framework decomposes retrieval into document and evidence selection, then marginalizes over top-m context candidates to perform reasoning.

Key innovations include:

  • Joint Modeling: The retriever and reader are trained end-to-end, allowing the reasoning component to provide feedback on which snippets are truly useful for answering the question.
  • Set-Valued Retrieval: Retrieval is split into document selection and snippet selection. A NULL token allows the model to reject irrelevant documents, which is crucial for identifying unanswerable questions.
  • Marginalization Objective: The training objective calculates the probability of the correct answer by marginalizing over the top-$m$ retrieved contexts. This rewards the model if any of the retrieved sets lead to the correct answer.
  • Heuristic Filtering and Auxiliary Loss: To prevent "lucky guesses" from noisy contexts, heuristics verify the validity of retrieved snippets. "Invalid" contexts are used as hard negatives to train the model to recognize unanswerable instances.

Experimental Results

We evaluated the model on two major benchmarks: IIRC, which focuses on multi-hop and numerical reasoning, and HotpotQA (fullwiki), which tests retrieval across millions of Wikipedia paragraphs.

Performance on the IIRC dataset. Our joint model (Ours) significantly outperforms pipeline baselines in both retrieval and reasoning metrics.

As shown in the results, the joint model with marginalization achieved a new state-of-the-art F1 score of 50.5 on IIRC, a significant improvement over standard pipeline approaches. On HotpotQA, the method yielded a 4.1-point QA F1 improvement over non-marginalized baselines.

Key Insights

  • Beyond Gold Labels: In the IIRC dataset, over 50% of questions have valid alternative evidence. By moving beyond rigid gold-label matching, our model generalizes better to real-world redundancy.
  • Robustness to Answer Types: The most significant gains were observed in "unanswerable" (+12.6 F1) and "Yes/No" (+10.2 F1) questions. These types often lack direct word overlap with the answer, making the reader-guided retrieval signal particularly valuable.
  • Retrieval vs. Accuracy: Interestingly, the model sometimes shows lower scores on traditional retrieval metrics (which only count gold matches) while achieving higher final QA accuracy. This confirms that the model is successfully utilizing valid, unlabeled evidence.

Conclusion

By treating retrieval as a latent process and marginalizing over potential evidence sets, we can mitigate the impact of incomplete annotations in MDQA. This approach not only improves overall accuracy but also makes systems more robust to the complexities of large-scale, redundant information sources. Future work will explore reducing the computational cost of joint training and refining heuristics for more abstract reasoning tasks.

Summary updated

BibTeX
@inproceedings{ni2021mitigating,
  author = {Ansong Ni and Matt Gardner and Pradeep Dasigi},
  title = {{Mitigating False-Negative Contexts in Multi-document Question Answering with Retrieval Marginalization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.497},
}
Figure