A Simple Yet Strong Pipeline for HotpotQA.

Dirk Groeneveld, Tushar Khot, Mausam, Ashish Sabharwal

Abstract

State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition. However, does their strong performance on popular multi-hop datasets really justify this added design complexity? Our results suggest that the answer may be no, because even our simple pipeline based on BERT, named Quark, performs surprisingly well. Specifically, on HotpotQA, Quark outperforms these models on both question answering and support identification (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially relevant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of Quark resurfaces the importance of carefully exploring simple model designs before using popular benchmarks to justify the value of complex techniques.

Introduction

Multi-hop Question Answering (QA) represents a significant challenge in natural language processing, requiring systems to synthesize information across multiple documents. Traditional approaches have leaned heavily on complex architectural additions, such as Graph Neural Networks (GNNs) and entity linking, under the assumption that explicit reasoning structures are necessary to bridge information gaps. This research challenges that assumption by introducing QUARK, a streamlined pipeline built on standard BERT components. The central inquiry is whether the added complexity of state-of-the-art models is truly justified by their performance, or if a simpler, well-designed pipeline can achieve comparable or superior results.

Methodology: The QUARK Pipeline

QUARK decomposes the multi-hop QA task into three distinct, manageable stages, leveraging the inherent power of transformer-based language models without specialized reasoning modules.

Overview of the QUARK three-step pipeline for multi-hop QA.
  1. Sentence Scoring ($r_{na}$): The system evaluates every sentence within the provided paragraphs independently. It calculates a relevance score for each sentence relative to the question. While sentences are scored independently, the model retains local paragraph context, which proves crucial for capturing implicit multi-hop links.
  2. Answer Prediction: The highest-ranked sentences are concatenated to form a consolidated context block. This block is then processed by a standard BERT-based span prediction model to extract the final answer.
  3. Support Identification ($r_a$): To satisfy the requirement for supporting facts, the sentence scorer is re-applied. In this pass, the predicted answer is included in the input, allowing the model to more accurately pinpoint the specific sentences that justify the chosen answer.

Experimental Results

The pipeline was evaluated on the HotpotQA benchmark, focusing on two primary settings: the Distractor setting (10 paragraphs provided) and the Fullwiki setting (open-domain retrieval from Wikipedia).

Performance on HotpotQA

QUARK demonstrates exceptional performance, particularly when compared to other BERT-based architectures that utilize more complex reasoning mechanisms.

Comparison of QUARK against state-of-the-art models on the HotpotQA Distractor set.

On the Distractor set, QUARK achieved an Answer F1 of 81.21 and a Joint F1 of 72.26, surpassing complex models like HGN and SAE. In the Fullwiki setting, QUARK maintained its lead among BERT-based models with a Joint F1 of 56.23. Notably, QUARK's performance is only eclipsed by models utilizing RoBERTa, which benefits from significantly larger training scales.

Discussion and Key Insights

The success of QUARK offers several key insights into the current state of multi-hop QA research:

  • The Power of Local Context: The ability to score sentences independently suggests that a strong language model can often resolve multi-hop dependencies if provided with sufficient local paragraph context. The "multi-hop" information is often implicitly handled by the language model's understanding of the paragraph.
  • Simplicity as a Reality Check: QUARK suggests that the gains attributed to complex reasoning modules (like GNNs or NER) may actually stem from the underlying transformer's capacity. It serves as a baseline to justify whether added complexity is truly necessary.
  • Iterative Refinement: Using the predicted answer to re-score sentences (Step 3) significantly improves the identification of supporting facts compared to doing it blindly.

Conclusion

While QUARK is highly effective, it is not without limitations. As a pipeline, it is susceptible to cascading errors; if the initial sentence selection fails, the subsequent QA stage cannot recover. Furthermore, its performance in open-domain settings is inherently limited by the quality of the initial paragraph retriever.

Ultimately, the strong performance of a simple BERT pipeline on HotpotQA suggests that current benchmarks may not strictly require the complex reasoning they were designed to test. Future research should focus on developing datasets that necessitate sophisticated multi-hop reasoning that cannot be bypassed by high-capacity language models using simple selection strategies.

Summary updated

Abstract

State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition. However, does their strong performance on popular multi-hop datasets really justify this added design complexity? Our results suggest that the answer may be no, because even our simple pipeline based on BERT, named Quark, performs surprisingly well. Specifically, on HotpotQA, Quark outperforms these models on both question answering and support identification (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially relevant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of Quark resurfaces the importance of carefully exploring simple model designs before using popular benchmarks to justify the value of complex techniques.
BibTeX
@inproceedings{groeneveld2020simple,
  author = {Dirk Groeneveld and Tushar Khot and Mausam and Ashish Sabharwal},
  title = {{A Simple Yet Strong Pipeline for HotpotQA.}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.711},
}
Figure