Abstract
Despite the rapid progress in multihop question-answering (QA), models still have trouble explaining why an answer is correct, with limited explanation training data available to learn from. To address this, we introduce three explanation datasets in which explanations formed from corpus facts are annotated. Our first dataset, eQASC, contains over 98K explanation annotations for the multihop question answering dataset QASC, and is the first that annotates multiple candidate explanations for each answer. The second dataset, eQASC-perturbed, is constructed by crowd-sourcing perturbations of a subset of explanations in QASC to test consistency and generalization. The third dataset, eOBQA, is constructed by adding explanation annotations to the OBQA dataset to test generalization. We show that this data can be used to significantly improve explanation quality (+14% absolute F1 over a strong retrieval baseline) using a BERT-based classifier. We also explore a delexicalized chain representation called Generalized Reasoning Chains (GRCs), finding that they maintain performance while being more robust to perturbations.
Introduction: The Explainability Gap in Multihop QA
While modern NLP models have become remarkably proficient at selecting the correct answer in multiple-choice settings, they often struggle to explain why that answer is correct. This is particularly evident in "multihop" question-answering, where a model must combine multiple disparate facts to reach a conclusion.
Existing research faces two major hurdles: a scarcity of data providing multiple valid explanation paths, and a reliance on "evident" reasoning structures where the path is hinted at in the question text. This paper addresses these gaps by focusing on harder cases where the reasoning structure is hidden—for example, knowing that "armor conducts electricity" requires first identifying that armor is made of metal, a fact not explicitly mentioned in the prompt.
Methods: Generalized Reasoning Chains (GRCs)
The researchers developed a BERT-based classifier to score the validity of reasoning chains. A reasoning chain is defined as a sequence of facts (e.g., $f_1$ AND $f_2$) that entails a conclusion $H$ (the question + the answer).
The core technical innovation is the Generalized Reasoning Chain (GRC). Instead of feeding raw text into the model, the chains are "variabilized" by replacing repeated noun phrases with variables ($X, Y, Z$). This abstraction forces the model to learn the underlying logic of the argument rather than just memorizing specific word associations.
As shown in the figure above, a specific instance regarding static electricity is abstracted into a template: "$X$ can cause $Y$ AND $Y$ can start $Z \rightarrow X$ can cause $Z$."
Datasets for Reasoning
The paper introduces three benchmarks to facilitate this research:
- eQASC: The primary dataset, containing over 98,000 crowd-annotated explanation candidates for the QASC dataset. It is unique because it annotates multiple candidate explanations (both valid and invalid) for every answer.
- eQASC-perturbed: A robustness test set where valid chains were modified using semantically invariant changes (e.g., changing "amphibians" to "frogs").
- eOBQA: A generalization set created by adding explanation annotations to the OpenBookQA dataset, used to test if models trained on eQASC can transfer their reasoning skills to a new domain.
Key Results and Insights
The study yielded several critical insights into how models handle logical structures:
- Performance Boost: The BERT-based classifier achieved a significant improvement over retrieval baselines. As detailed in the results table below, the BERT-CHAIN and BERT-GRC models significantly outperform standard retrieval and QA models in both classification (F1, AUC-ROC) and ranking (P@1, NDCG) metrics.
- Logic over Lexicon: By using GRCs, models maintained high performance while becoming much more stable. Standard models often flip their predictions if a single word is changed to a synonym; GRC models are largely immune to this because they focus on the underlying template.
- Robustness: On the perturbed dataset, the standard BERT-CHAIN model showed a change in prediction for almost every instance (only 0.23% consistency). In contrast, the BERT-GRC model remained consistent in 40.8% of cases.
Limitations and Future Work
Despite these gains, challenges remain. The authors noted that in 24% of cases, the initial retrieval system failed to find any valid reasoning chain in its top 10 results, suggesting that retrieval is a major bottleneck. Furthermore, this study focused primarily on 2-step reasoning chains. Real-world scientific or legal reasoning often requires much longer hops, which increases the complexity of both retrieval and variabilization. Future work may involve joint models that learn to retrieve and verify chains simultaneously to overcome the current retrieval ceiling.
Summary updated