Retrieval Data Augmentation Informed by Downstream Question Answering Performance

James Ferguson, Hannaneh Hajishirzi, Pradeep Dasigi, Tushar Khot2022 · DOI

Abstract

Training retrieval models to fetch contexts for Question Answering (QA) over large corpora requires labeling relevant passages. Since obtaining exhaustive manual annotations is infeasible, prior work often uses text overlap heuristics. However, these fail when tasks require deep reasoning or non-extractive answers. We address this by identifying relevant passages based on their utility to a trained QA model, using a search process guided by the model's loss. Our experiments show this approach identifies relevant context for unseen data over 90% of the time on the IIRC dataset and improves downstream QA performance on both IIRC and QASC.

Introduction

In open-domain Question Answering, systems must retrieve a small set of relevant documents from a massive corpus before "reading" them to extract or generate an answer. Training the retrieval component typically relies on manual annotations or distant supervision heuristics (e.g., assuming any passage containing the answer string is relevant). Both approaches are flawed: manual labels are often incomplete, and string-matching fails for complex reasoning tasks where the answer is not a simple text span, such as boolean or numeric questions.

This paper addresses the "label sparsity" problem by introducing a method to automatically discover alternative, relevant training passages. By identifying contexts that help a QA model minimize its loss, we can augment training data with diverse reasoning paths that human annotators might have missed.

The core innovation is using a QA model as a sophisticated relevance filter. The proposed pipeline consists of three stages:

  1. Base Model Training: A QA model is initially trained on existing "gold" (manually labeled) data.
  2. Context Discovery: For each training question, the system searches the corpus for passages that result in the lowest model loss when processed by the base QA model. These are passages that make the model most confident in the correct answer.
  3. Augmentation and Retraining: These discovered passages are added to the training set. A new retrieval model is then trained on this augmented data, followed by a final QA model trained on the improved retriever's outputs.

To ensure robustness and prevent memorization, the authors employ a 10-fold cross-labeling approach, where models trained on 90% of the data are used to label the remaining 10%.

Figure 1: Qualitative comparison of contexts identified by the QA-loss method versus standard BM25 retrieval. The QA-loss approach successfully identifies relevant reasoning paths that keyword matching misses.

Key Insights: Label Sparsity and Viable Paths

A critical finding of this work is the extent of label sparsity in existing datasets. Analysis revealed that approximately 70% of questions in the QASC dataset have multiple viable reasoning paths to the correct answer, yet traditional "gold" labels capture only a fraction of these. By defining relevance through model utility (loss minimization), the system can find useful contexts for non-extractive tasks where traditional heuristics are inapplicable.

Experimental Results

The QA-loss informed augmentation significantly outperformed standard baselines. In terms of retrieval accuracy during the search process, the method identified relevant contexts 91% of the time on the IIRC dataset, compared to only 38% for BM25.

Figure 2: Quantitative analysis of the QA-loss method. (a) Retrieval accuracy compared to baselines. (b) Performance across different question types. (c) Distribution of viable contexts per question, highlighting the label sparsity problem.

Downstream QA performance also saw notable gains:

  • IIRC: The final model achieved a 0.5 F1 point improvement over the supervised baseline.
  • QASC: Accuracy increased by 2.1 points, reaching a final score of 73.9%.

These results suggest that exposing the retriever to alternative reasoning paths helps it learn more generalizable logic rather than just memorizing specific "gold" facts.

Limitations and Future Work

Despite its success, the model occasionally relies on "shortcuts"—selecting passages that contain the final answer (e.g., a total count) without providing the necessary intermediate reasoning steps. It also struggled with "Span Compare" questions, where it might find the context for the correct answer but fail to retrieve the context needed for the comparison. Future research will focus on ensuring the retrieval of complete reasoning chains.

Summary updated

Abstract

Training retrieval models to fetch contexts for Question Answering (QA) over large corpora requires labeling relevant passages in those corpora. Since obtaining exhaustive manual annotations of all relevant passages is not feasible, prior work uses text overlap heuristics to find passages that are likely to contain the answer, but this is not feasible when the task requires deeper reasoning and answers are not extractable spans (e.g.: multi-hop, discrete reasoning). We address this issue by identifying relevant passages based on whether they are useful for a trained QA model to arrive at the correct answers, and develop a search process guided by the QA model's loss. Our experiments show that this approach enables identifying relevant context for unseen data greater than 90% of the time on the IIRC dataset and generalizes better to the end QA task than those trained on just the gold retrieval data on IIRC and QASC datasets.
BibTeX
@misc{ferguson2022retrieval,
  author = {James Ferguson and Hannaneh Hajishirzi and Pradeep Dasigi and Tushar Khot},
  title = {{Retrieval Data Augmentation Informed by Downstream Question Answering Performance}},
  year = {2022},
  doi = {10.18653/v1/2022.fever-1.1},
}
Figure