Abstract
Learning multi-hop reasoning has been a key challenge for reading comprehension models, leading to the design of datasets that explicitly focus on it. Ideally, a model should not be able to perform well on a multi-hop question answering task without doing multi-hop reasoning. In this paper, we investigate two recently proposed datasets, WikiHop and HotpotQA. First, we explore sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets. Furthermore, we find spurious correlations in the unmasked version of WikiHop, which make it easy to achieve high performance considering only the questions and answers. Finally, we investigate one key difference between these datasets, namely span-based vs. multiple-choice formulations of the QA task. Multiple-choice versions of both datasets can be easily gamed, and two models we examine only marginally exceed a baseline in this setting. Overall, while these datasets are useful testbeds, high-performing models may not be learning as much multi-hop reasoning as previously thought.
Introduction
In Natural Language Processing, "multi-hop reasoning" refers to the ability of a model to connect multiple pieces of information across different parts of a text to answer a complex question. For instance, answering "Where was the father of the 44th US President born?" requires identifying the father (Barack Obama Sr.) and then locating his birthplace (Kenya). Large-scale datasets like WikiHop and HotpotQA were specifically designed to benchmark this capability. However, recent trends in deep learning suggest that models often exploit statistical "shortcuts" to achieve high scores without truly understanding the underlying logic. This study investigates whether these benchmarks are truly testing multi-hop reasoning or if they are vulnerable to simpler pattern-matching strategies.
Probing for Single-Hop Shortcuts
To determine if multi-hop reasoning is strictly necessary for these datasets, the authors employed sentence-factored models. These models, such as a modified BiDAF, are architecturally restricted to scoring each sentence in a passage independently. Because they cannot integrate information across sentences, any success they achieve indicates that the question can be solved using "single-hop" logic—often because the answer sentence contains enough lexical overlap with the question to be identified directly.
As shown in the results, the Factored BiDAF model achieved an accuracy of 66.1% on WikiHop and 57.2% on HotpotQA. This suggests that for a significant portion of these datasets, the intended multi-hop requirement is optional, as the models can find the correct answer by focusing on individual sentences.
The Impact of Task Formulation: Multiple-Choice vs. Span Extraction
A critical finding of this research is that the format of the question—multiple-choice (MC) versus span-based extraction—drastically affects how easily a dataset can be "gamed."
In WikiHop, which uses a multiple-choice format, a No-Context baseline (which only sees the question and candidate answers) reached nearly 60% accuracy. This indicates that the model learned to exploit "entity typing" and other spurious correlations within the candidate list. For example, if a question asks for an employer, the model may simply select the candidate that resembles an organization, ignoring the supporting text entirely.
To further test this, the researchers recast HotpotQA (originally span-based) into a multiple-choice task. They found that the "cheatability" of the dataset increased significantly in the MC format. Interestingly, a model trained on span-based data and then adapted to multiple-choice (Span2MC) performed better than models trained natively on the multiple-choice data.
Conclusion and Recommendations
The study concludes that while WikiHop and HotpotQA remain valuable benchmarks, high leaderboard scores should be interpreted with caution. The authors recommend that dataset creators use sentence-factored and no-context baselines during the design phase to filter out questions that do not strictly require multi-hop reasoning. Furthermore, the results strongly suggest that span-based supervision is more robust than multiple-choice formats, as it forces models to learn more meaningful features from the text rather than exploiting correlations in a candidate list. Future work should focus on datasets that require deeper semantic entailment, moving beyond simple extractive pattern matching.
Summary updated