Performance Prediction for Multi-hop Questions

Mohammadreza Samadi, Davood RafieiarXiv 2023 · DOI · arXiv

Introduction

Query Performance Prediction (QPP) is a critical component of information retrieval, aimed at estimating the quality of search results before or after retrieval. While existing QPP models are effective for single-hop questions (e.g., "Who is the President of France?"), they struggle with the increasing complexity of multi-hop Question Answering (QA). Multi-hop questions require connecting information across multiple documents, creating a "scavenger hunt" where a failure at any step can cause the entire chain to collapse.

Traditional QPP models often treat queries as a "bag of words," failing to account for the structural dependencies and reasoning required to bridge documents. To address this, researchers have developed multHP, the first pre-retrieval method specifically designed to predict the difficulty of multi-hop questions by modeling the search process as a structured retrieval path.

Modeling Retrieval Paths

The core innovation of multHP is the formalization of multi-hop questions into distinct Retrieval Paths. By identifying the structural relationship between the question and the necessary supporting documents, the system can estimate performance without fetching a single document.

The authors categorize these paths into three primary types:

  1. Bridge: A sequential dependency where the second document can only be identified after the first is retrieved.
  2. Comparison: Both documents can be identified independently from the question text (e.g., comparing attributes of two entities).
  3. Mixed: A combination of sequential and independent retrieval paths.
Retrieval path structures for multi-hop questions: (a) Bridge, (b) Comparison, and (c) Mixed.

How multHP Works

Operating in a pre-retrieval setting, multHP uses question text and corpus statistics to estimate difficulty:

  • Entity and Phrase Extraction: The model identifies named entities and "frozen phrases" (like movie titles) within the query.
  • Probability Estimation: Using n-gram counts from the corpus, it calculates the probability $P(c|q)$ of finding a specific document. Rare entities generally lead to higher success probabilities.
  • Path-Based Aggregation: These probabilities are combined according to the path type. For "Bridge" questions, success depends on the product of finding both hops sequentially, capturing the "weakest link" bottleneck.
Examples of Bridge and Comparison questions from the HotpotQA dataset.

Experimental Evaluation

The researchers evaluated multHP primarily on HotpotQA, a benchmark containing over 113,000 multi-hop question-answer pairs. They also tested generalization on WikiPassageQA and WikiQA (single-hop).

Key Results

  • Superior Correlation: multHP significantly outperformed traditional QPP baselines (such as SCS, maxIDF, and maxSCQ) across both sparse (GoldEn) and dense (MDR) retrievers.
  • Bridge Performance: On challenging "Bridge" questions in HotpotQA, multHP achieved a Pearson correlation of 0.2342—nearly double the 0.1418 achieved by the best baseline.
  • Adaptive Retrieval: The model enables "Adaptive Retrieval," where the system fetches more documents (increasing $k$) for questions predicted to be difficult. This strategy improves F1-scores under fixed computational budgets compared to using a constant $k$.

Insights and Future Directions

The success of multHP highlights that in multi-hop QA, the relationship between terms (the "hop") is a more significant source of failure than keyword rarity alone. By modeling the compounding difficulty of sequential steps, multHP provides a more realistic estimate of system performance.

However, limitations remain. The model's reliance on entity extraction makes it sensitive to ambiguous phrasing. Future work may involve incorporating semantic embeddings to bridge the "semantic gap" where questions and documents use different terminology for the same concepts, as well as extending path modeling to handle more complex reasoning types like negation or intersection.

Summary updated

Abstract

We study the problem of Query Performance Prediction (QPP) for open-domain multi-hop Question Answering (QA), where the task is to estimate the difficulty of evaluating a multi-hop question over a corpus. Despite the extensive research on predicting the performance of ad-hoc and QA retrieval models, there has been a lack of study on the estimation of the difficulty of multi-hop questions. The problem is challenging due to the multi-step nature of the retrieval process, potential dependency of the steps and the reasoning involved. To tackle this challenge, we propose multHP, a novel pre-retrieval method for predicting the performance of open-domain multi-hop questions. Our extensive evaluation on the largest multi-hop QA dataset using several modern QA systems shows that the proposed model is a strong predictor of the performance, outperforming traditional single-hop QPP models. Additionally, we demonstrate that our approach can be effectively used to optimize the parameters of QA systems, such as the number of documents to be retrieved, resulting in improved overall retrieval performance.
BibTeX
@misc{samadi2023performance,
  author = {Mohammadreza Samadi and Davood Rafiei},
  title = {{Performance Prediction for Multi-hop Questions}},
  year = {2023},
  eprint = {2308.06431},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.06431},
}
Figure