Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension

Naoya Inoue, Harsh Trivedi, Steven Sinha, Niranjan Balasubramanian, Kentaro Inui

Introduction

In the field of multi-hop Reading Comprehension (RC), AI systems are required to synthesize information across multiple documents to arrive at a correct answer. While achieving high accuracy is a primary goal, making these systems "explainable" is equally vital for human trust and utility. Traditional approaches rely on extractive methods—simply highlighting existing sentences from the source text. However, these often fail to provide clear, concise reasoning.

This paper introduces SuQA (Summarizer-augmented QA), a framework that shifts the paradigm from extraction to abstractive summarization. By generating new, condensed explanations, SuQA provides justifications that are significantly more compact and human-readable without sacrificing the information necessary to answer complex questions.

The Limitations of Extractive Explanations

Current explainable RC models typically perform extractive sentence selection. While straightforward, this method suffers from three critical flaws:

  1. Redundancy: Extractive models often include irrelevant details (e.g., middle names, dates, or tangential facts) that clutter the explanation.
  2. Contextual Dependency: Selected sentences frequently contain unresolved pronouns like "he" or "it," which are incomprehensible when stripped of their surrounding text.
  3. Incompleteness: The logical "bridge" between two disparate facts is often implied rather than explicitly stated in a single sentence, making simple extraction insufficient for complex reasoning.
Comparison between extractive (left) and abstractive (right) explainers. SuQA generates a concise 21-word summary compared to the 77-word extractive baseline.

The authors argue that an ideal explanation should be concise: minimal in length, sufficient to support the answer, and linguistically comprehensible.

The SuQA Framework

SuQA addresses these issues through a "summarize-then-answer" pipeline consisting of two primary modules:

  1. Abstractive Explainer (AX): Built on the BART architecture, this module generates a condensed summary of relevant facts instead of merely selecting sentences. It is designed to paraphrase and resolve coreferences to ensure clarity.
  2. QA Module (QAM): This component receives only the generated summary to produce the final answer. This setup ensures "faithfulness"—if the QAM can successfully answer the question using only the summary, the explanation is proven to be sufficient.
The SuQA system architecture, illustrating the interaction between the Abstractive Explainer, the QA Model, and the Reinforcement Learning reward loop.

Training via Reinforcement Learning

A significant challenge in developing abstractive explainers is the scarcity of human-annotated datasets. To overcome this, the researchers employed a semi-supervised approach. After initial pre-training on general summarization and a small set of human-labeled explanations (from the R4C dataset), the model was fine-tuned using Reinforcement Learning (RL).

The RL reward function was designed as a geometric mean of three key metrics:

  • Conciseness: Encouraging higher compression ratios to strip away "fluff."
  • Sufficiency: Rewarding the system when the QA module correctly answers the question based on the summary.
  • Comprehensibility: Utilizing a linguistic acceptability scorer to ensure the generated text is natural and grammatically correct.

Key Results and Insights

Evaluation on the HotpotQA dataset revealed that SuQA significantly outperforms extractive baselines in qualitative metrics:

  • Superior Compression: SuQA achieved a compression rate 2.9x higher than traditional extractive methods.
  • Enhanced Sufficiency: Despite the drastic reduction in word count, human evaluators rated SuQA explanations 2.5 points higher in sufficiency compared to extractive ones.
  • Low-Data Efficiency: The RL training regime proved highly effective; with only 298 human-annotated samples, human-judged sufficiency improved by +11.0 points.

Furthermore, the researchers identified a "Disconnected Reasoning" trap in extractive models. Large models often rely on "shortcut" lexical cues in long paragraphs to find answers. By forcing the model to operate on a tiny, abstractive summary, SuQA ensures the system actually utilizes the logical connections required for multi-hop reasoning.

Conclusion

SuQA demonstrates that AI can be a more effective "teacher" by paraphrasing information rather than just quoting it. While there is a slight trade-off in raw F1 scores—likely due to the removal of the aforementioned "shortcuts"—the gains in explanation quality and faithfulness mark a significant step forward for interpretable AI in complex reading tasks. Future work will look toward jointly optimizing the explainer and QA modules to further bridge the performance gap.

Summary updated

Abstract

How can we generate concise explanations for multi-hop Reading Comprehension (RC)? The current strategies of identifying supporting sentences can be seen as an extractive question-focused summarization of the input text. However, these extractive explanations are not necessarily concise i.e. not minimally sufficient for answering a question. Instead, we advocate for an abstractive approach, where we propose to generate a question-focused, abstractive summary of input paragraphs and then feed it to an RC system. Given a limited amount of human-annotated abstractive explanations, we train the abstractive explainer in a semi-supervised manner, where we start from the supervised model and then train it further through trial and error maximizing a conciseness-promoted reward function. Our experiments demonstrate that the proposed abstractive explainer can generate more compact explanations than an extractive explainer with limited supervision (only 2k instances) while maintaining sufficiency.
BibTeX
@inproceedings{inoue2021summarize,
  author = {Naoya Inoue and Harsh Trivedi and Steven Sinha and Niranjan Balasubramanian and Kentaro Inui},
  title = {{Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.490},
}
Figure