TL;DR
State-of-the-art multi-hop question answering (QA) currently relies on "dense" retrieval models that are massive GPU hogs and often blind to simple keyword matches. This paper introduces a hybrid approach—combining traditional lexical reranking with dense retrieval—that achieves competitive performance while requiring eight times less computational power than fully dense models.
What Problem Does This Paper Address?
In open-domain QA, a system must find relevant passages from a massive collection (like Wikipedia) to answer a user's question. "Multi-hop" QA is particularly tricky because the answer isn't in a single document; the system must find one passage, extract a "bridge" entity, and use that to find a second passage containing the final answer.
Currently, Dense Passage Retrieval (DPR) is the gold standard. It uses neural networks to represent questions and passages as high-dimensional vectors. However, these models have two major drawbacks:
- Resource Intensity: Training them requires massive GPU clusters to handle large batch sizes, which are necessary for "in-batch negative" training.
- Lexical Blindness: They often struggle when a question has high lexical overlap (shared words) with a passage, ironically failing where simpler keyword-based methods (like BM25) excel.
The authors set out to see if they could build a retriever that is both computationally "cheap" and more robust by combining the strengths of lexical and dense methods.
Methods
The authors propose a hybrid retriever called Rerank + DPR2. Instead of using a single complex neural architecture for both retrieval steps, they split the task:
- Hop 1 (Lexical + Rerank): They use BM25 to find candidate passages based on keyword matching, then use a BERT-based classifier to rerank the top 100 results. This captures the high lexical overlap often found in the first step of a multi-hop question.
- Hop 2 (Dense): For the second hop, they train a specific DPR model ($DPR_2$). This model takes the question concatenated with the passage found in Hop 1 as input to retrieve the final relevant passage.
By using a standard reranker for the first hop, they drastically reduce the training time and hardware requirements compared to fully dense iterative models like MDR (Multihop Dense Retrieval).
Datasets
The study uses HotpotQA in the "full-wiki" setting. This is a benchmark dataset containing ~113k questions that require two hops of reasoning across all of Wikipedia. Questions are categorized as:
- Bridge: Where a missing entity connects two passages.
- Comparison: Where two entities are mentioned and compared (e.g., "Which is smaller, X or Y?").
Key Insights and Ideas
- The Batch Size Trap: Dense models rely on in-batch negatives—using other positive passages in the same training batch as "wrong" examples. On limited hardware (e.g., a single 12GB GPU), batch sizes are forced to be small, which starves the model of difficult negative examples and causes performance to plummet.
- Lexical Overlap is a Feature, Not a Bug: While researchers have moved toward "semantic" dense retrieval, many multi-hop questions contain specific names or terms that appear exactly in the source text. Lexical models capture these "n-gram matches" perfectly, whereas dense models might get distracted by semantically similar but irrelevant passages.
- Efficiency as a First-Class Citizen: The authors demonstrate that a hybrid model trained on a single consumer-grade GPU can outperform a SOTA dense model trained on the same hardware and remain competitive with models trained on 8x 32GB GPU clusters.
Key Results
The hybrid Rerank + DPR2 model showed significant gains in low-resource settings. As seen in the performance breakdown, the hybrid approach significantly outperforms the standard MDR model when both are restricted to limited hardware.
Specifically:
- On a single 12GB GPU, Rerank + DPR2 achieved an EM@20 of 0.762, while the dense MDR model only reached 0.619.
- The hybrid model is highly competitive with the "Full" MDR model, which requires 8x 32GB GPUs and significantly more training time.
The authors also analyzed how performance degrades as hardware is restricted. Table 2 shows that for purely dense models, there is a direct, steep correlation between GPU count/batch size and retrieval accuracy.
Limitations and Future Work
The main limitation of the hybrid approach occurs when lexical overlap is misleading. If the question contains keywords that appear in the second passage but not the first, the Hop 1 lexical retriever might skip the necessary "bridge" passage entirely, leading the system down a dead end.
For future work, the authors plan to explore more sophisticated ways to combine lexical and dense strengths, potentially creating models that can dynamically decide which retrieval strategy to use based on the question's characteristics.
Summary updated