Multi-hop Question Answering via Reasoning Chains

Jifan Chen, Shih-Ting Lin, Greg DurrettarXiv 2019 · OpenReview

Abstract

Multi-hop question answering requires models to gather information from different parts of a text to answer a question. Most current approaches learn to address this task in an end-to-end way with neural networks, without maintaining an explicit representation of the reasoning process. We propose a method to extract a discrete reasoning chain over the text, which consists of a series of sentences leading to the answer. We then feed the extracted chains to a BERT-based QA model to do final answer prediction. Critically, we do not rely on gold annotated chains or "supporting facts:" at training time, we derive pseudogold reasoning chains using heuristics based on named entity recognition and coreference resolution. Nor do we rely on these annotations at test time, as our model learns to extract chains from raw text alone. We test our approach on two recently proposed large multi-hop question answering datasets: WikiHop and HotpotQA, and achieve state-of-art performance on WikiHop and strong performance on HotpotQA. Our analysis shows the properties of chains that are crucial for high performance: in particular, modeling extraction sequentially is important, as is dealing with each candidate sentence in a context-aware way. Furthermore, human evaluation shows that our extracted chains allow humans to give answers with high confidence, indicating that these are a strong intermediate abstraction for this task.

Introduction

Multi-hop question answering (QA) presents a significant challenge: a model must aggregate disparate pieces of information across multiple documents or paragraphs to arrive at a correct answer. For instance, answering "What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell?" requires first identifying the actress (Shirley Temple) and then locating her specific government roles in a separate context.

Example of multi-hop reasoning chains connecting a question to an answer across multiple documents.

Traditional end-to-end neural models often act as "black boxes," making it difficult to discern if they are truly reasoning or merely exploiting lexical shortcuts. This lack of interpretability can lead to fragile performance. This paper introduces a framework that explicitly extracts "reasoning chains"—sequences of sentences that logically bridge the gap between the question and the answer—before performing final prediction.

Methodology

The proposed system follows a two-stage pipeline: Chain Extraction and Answer Prediction.

1. Chain Extraction

The goal of the first stage is to identify a sequence of sentences that form a logical path. Rather than treating sentences as an unordered set, the model views them as nodes in a graph where edges represent shared entities or paragraph co-occurrence.

The model architecture consisting of a BERT-based context encoder and an RNN-based sentence extractor.

To train this extractor without human-labeled paths, the authors developed "pseudogold" chains using heuristics like Shortest Path (minimizing hops) and Question Overlap. The extractor itself uses BERT-Para, which encodes paragraphs alongside the question to maintain local context. A Recurrent Neural Network (RNN) with a pointer mechanism then sequentially selects sentences to construct the chain.

2. Answer Prediction

The extracted chains—specifically the top-ranked sequences from a beam search—are concatenated and passed to a BERT-based reader. By pruning the vast search space into a concise chain of relevant sentences, the reader can focus on the most critical information, improving both accuracy and computational efficiency.

Experimental Results

The model was evaluated on two prominent multi-hop benchmarks: WikiHop and HotpotQA.

  • WikiHop: The model achieved 76.5% accuracy on the test set, establishing a new state-of-the-art at the time. This outperformed complex graph-based architectures, demonstrating the power of sequential chain extraction.
  • HotpotQA: Despite not using the dataset's provided "supporting fact" labels during training, the model reached a strong F1 score of 74.11. This highlights the robustness of the heuristic-based training approach.

Key Insights

  • Sequentiality is Key: Extracting sentences as an ordered path is significantly more effective than selecting them as an unordered set. Reasoning is fundamentally a directed process.
  • Contextual Encoding: Encoding sentences within their paragraph context (BERT-Para) is superior to encoding them in isolation. Local cues, such as coreference, are essential for linking sentences.
  • Human Interpretability: Human evaluation confirmed that the extracted chains are highly informative. Participants could answer questions with high confidence using only the model-provided chains, suggesting the model's internal logic aligns with human reasoning.

Conclusion

By forcing a model to produce explicit reasoning chains, we gain both performance and transparency. This approach proves that sophisticated multi-hop reasoning can be learned through simple heuristics and sequential extraction, providing a scalable path toward more interpretable and reliable AI systems.

Summary updated

Abstract

Multi-hop question answering requires models to gather information from different parts of a text to answer a question. Most current approaches learn to address this task in an end-to-end way with neural networks, without maintaining an explicit representation of the reasoning process. We propose a method to extract a discrete reasoning chain over the text, which consists of a series of sentences leading to the answer. We then feed the extracted chains to a BERT-based QA model to do final answer prediction. Critically, we do not rely on gold annotated chains or "supporting facts:" at training time, we derive pseudogold reasoning chains using heuristics based on named entity recognition and coreference resolution. Nor do we rely on these annotations at test time, as our model learns to extract chains from raw text alone. We test our approach on two recently proposed large multi-hop question answering datasets: WikiHop and HotpotQA, and achieve state-of-art performance on WikiHop and strong performance on HotpotQA. Our analysis shows the properties of chains that are crucial for high performance: in particular, modeling extraction sequentially is important, as is dealing with each candidate sentence in a context-aware way. Furthermore, human evaluation shows that our extracted chains allow humans to give answers with high confidence, indicating that these are a strong intermediate abstraction for this task.
BibTeX
@misc{chen2019multi,
  author = {Jifan Chen and Shih-Ting Lin and Greg Durrett},
  title = {{Multi-hop Question Answering via Reasoning Chains}},
  year = {2019},
  eprint = {1910.02610},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.02610},
}
Figure