Combining Lexical and Dense Retrieval for Computationally Efficient Multi-hop Question Answering

Georgios Sidiropoulos, Nikos Voskarides, Svitlana Vakulenko, Evangelos KanoulasarXiv 2021 · arXiv

TL;DR

State-of-the-art multi-hop question answering (QA) currently relies on "dense" retrieval models that are massive GPU hogs and often blind to simple keyword matches. This paper introduces a hybrid approach—combining traditional lexical reranking with dense retrieval—that achieves competitive performance while requiring eight times less computational power than fully dense models.

What Problem Does This Paper Address?

In open-domain QA, a system must find relevant passages from a massive collection (like Wikipedia) to answer a user's question. "Multi-hop" QA is particularly tricky because the answer isn't in a single document; the system must find one passage, extract a "bridge" entity, and use that to find a second passage containing the final answer.

Currently, Dense Passage Retrieval (DPR) is the gold standard. It uses neural networks to represent questions and passages as high-dimensional vectors. However, these models have two major drawbacks:

  1. Resource Intensity: Training them requires massive GPU clusters to handle large batch sizes, which are necessary for "in-batch negative" training.
  2. Lexical Blindness: They often struggle when a question has high lexical overlap (shared words) with a passage, ironically failing where simpler keyword-based methods (like BM25) excel.

The authors set out to see if they could build a retriever that is both computationally "cheap" and more robust by combining the strengths of lexical and dense methods.

Methods

The authors propose a hybrid retriever called Rerank + DPR2. Instead of using a single complex neural architecture for both retrieval steps, they split the task:

  • Hop 1 (Lexical + Rerank): They use BM25 to find candidate passages based on keyword matching, then use a BERT-based classifier to rerank the top 100 results. This captures the high lexical overlap often found in the first step of a multi-hop question.
  • Hop 2 (Dense): For the second hop, they train a specific DPR model ($DPR_2$). This model takes the question concatenated with the passage found in Hop 1 as input to retrieve the final relevant passage.

By using a standard reranker for the first hop, they drastically reduce the training time and hardware requirements compared to fully dense iterative models like MDR (Multihop Dense Retrieval).

Datasets

The study uses HotpotQA in the "full-wiki" setting. This is a benchmark dataset containing ~113k questions that require two hops of reasoning across all of Wikipedia. Questions are categorized as:

  • Bridge: Where a missing entity connects two passages.
  • Comparison: Where two entities are mentioned and compared (e.g., "Which is smaller, X or Y?").

Key Insights and Ideas

  • The Batch Size Trap: Dense models rely on in-batch negatives—using other positive passages in the same training batch as "wrong" examples. On limited hardware (e.g., a single 12GB GPU), batch sizes are forced to be small, which starves the model of difficult negative examples and causes performance to plummet.
  • Lexical Overlap is a Feature, Not a Bug: While researchers have moved toward "semantic" dense retrieval, many multi-hop questions contain specific names or terms that appear exactly in the source text. Lexical models capture these "n-gram matches" perfectly, whereas dense models might get distracted by semantically similar but irrelevant passages.
  • Efficiency as a First-Class Citizen: The authors demonstrate that a hybrid model trained on a single consumer-grade GPU can outperform a SOTA dense model trained on the same hardware and remain competitive with models trained on 8x 32GB GPU clusters.

Key Results

The hybrid Rerank + DPR2 model showed significant gains in low-resource settings. As seen in the performance breakdown, the hybrid approach significantly outperforms the standard MDR model when both are restricted to limited hardware.

Overall retrieval performance comparing single-hop, multi-hop, and the proposed hybrid model.

Specifically:

  • On a single 12GB GPU, Rerank + DPR2 achieved an EM@20 of 0.762, while the dense MDR model only reached 0.619.
  • The hybrid model is highly competitive with the "Full" MDR model, which requires 8x 32GB GPUs and significantly more training time.

The authors also analyzed how performance degrades as hardware is restricted. Table 2 shows that for purely dense models, there is a direct, steep correlation between GPU count/batch size and retrieval accuracy.

Analysis of how computational resources affect the performance of dense models.

Limitations and Future Work

The main limitation of the hybrid approach occurs when lexical overlap is misleading. If the question contains keywords that appear in the second passage but not the first, the Hop 1 lexical retriever might skip the necessary "bridge" passage entirely, leading the system down a dead end.

Table 3: Two example bridge questions for which Rerank+DPR2 retrieves both relevant passages at the top posi- tions while MDR fails to do so. Lexical overlap is indicated with underlined text while the answer to the question is highlighted.

For future work, the authors plan to explore more sophisticated ways to combine lexical and dense strengths, potentially creating models that can dynamically decide which retrieval strategy to use based on the question's characteristics.

Summary updated

Abstract

In simple open-domain question answering (QA), dense retrieval has become one of the standard approaches for retrieving the relevant passages to infer an answer. Recently, dense retrieval also achieved state-of-the-art results in multi-hop QA, where aggregating information from multiple pieces of information and reasoning over them is required. Despite their success, dense retrieval methods are computationally intensive, requiring multiple GPUs to train. In this work, we introduce a hybrid (lexical and dense) retrieval approach that is highly competitive with the state-of-the-art dense retrieval models, while requiring substantially less computational resources. Additionally, we provide an in-depth evaluation of dense retrieval methods on limited computational resource settings, something that is missing from the current literature.
BibTeX
@misc{sidiropoulos2021combining,
  author = {Georgios Sidiropoulos and Nikos Voskarides and Svitlana Vakulenko and Evangelos Kanoulas},
  title = {{Combining Lexical and Dense Retrieval for Computationally Efficient Multi-hop Question Answering}},
  year = {2021},
  eprint = {2106.08433},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.08433},
}

Loading related content...

Figure