Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering

Vikas Yadav, Steven Bethard, Mihai Surdeanu

TL;DR

This paper introduces ROCC, a novel unsupervised method for selecting crucial sentences (justifications) that help answer complex, multi-hop questions. By balancing relevance, diversity, and coverage, ROCC significantly boosts the performance of state-of-the-art question-answering systems on challenging datasets, achieving new state-of-the-art results without requiring any labeled justification data.

What Problem Does This Paper Address?

Interpretable AI is crucial for widespread adoption, especially in complex tasks like question answering (QA). While many QA systems achieve high accuracy, understanding how they arrive at an answer is often difficult. For multi-hop QA, where answers require synthesizing information from multiple sources, providing human-readable explanations (justifications) is key to interpretability. However, automatically selecting the right justification sentences is challenging. Existing methods often rely on expensive, labeled data or struggle with diverse knowledge sources. This paper addresses the need for an unsupervised, robust, and effective method to select these critical justification sentences for multi-hop QA.

Methods

The core contribution is the ROCC (Relevance, Overlap, Coverage, and Coherence) algorithm. ROCC operates in several steps:

  1. Retrieval of Candidate Sentences: For datasets with large knowledge bases (like ARC), it uses BM25 (a standard information retrieval model) to retrieve the top N sentences based on a query combining the question and candidate answer. For reading comprehension datasets (like MultiRC), all sentences in the provided passage are candidates.
  2. Generation of Candidate Justification Sets: Instead of ranking individual sentences, ROCC considers sets of sentences. It generates multiple candidate sets by varying the number of sentences (k) from the retrieved candidates.
  3. Ranking of Candidate Justification Sets: Each candidate set is scored using the ROCC formula, which balances three criteria:
  4. Relevance (R): How relevant are the sentences to the question and answer, measured by the average BM25 score.
  5. Overlap (O): Minimizing lexical overlap between sentences in a set to ensure diversity and avoid redundancy.
  6. Coverage (C): Maximizing the lexical coverage of the question and answer terms, weighted by term importance (IDF). The set with the highest ROCC score is selected as the justification.

  7. Answer Classification: The selected justification sentences are then fed, along with the question and candidate answer, into a supervised QA model (specifically, a BERT-based classifier). For MultiRC, where sentences are from the same passage, they are concatenated. For ARC, where sentences might come from different sources, they are processed individually and their scores averaged.

A key aspect is that ROCC is unsupervised, meaning it doesn't require any labeled justification data for training.

Datasets

The paper evaluates ROCC on two prominent multi-hop QA datasets:

  • AI2's Reasoning Challenge (ARC): A multiple-choice QA dataset with questions from science exams, featuring a large knowledge base of unstructured text.
  • Multi-Sentence Reading Comprehension (MultiRC): A reading comprehension dataset where questions are accompanied by a supporting passage containing the answer.

Key Insights and Ideas

  • Unsupervised Justification Selection is Viable: The paper demonstrates that effective justification selection can be achieved without relying on labeled data, which is a significant practical advantage.
  • Balancing Multiple Criteria is Crucial: The ROCC formula's success stems from its careful balance of relevance, diversity (minimizing overlap), and comprehensiveness (maximizing coverage). This holistic approach is more effective than simply picking the most relevant sentences.
  • Sets, Not Just Sentences: Treating justifications as sets rather than individual sentences allows for better knowledge aggregation and coherence.
  • Robustness Across Domains: ROCC shows remarkable stability across different domains within a dataset, outperforming supervised methods that are often domain-specific. This makes it more practical for real-world applications where data distributions can shift.
  • Interpretability Boost: By providing better justifications, ROCC not only improves QA performance but also enhances the interpretability of the overall system.

Key Results

  • State-of-the-Art Performance: ROCC achieved new state-of-the-art results on both ARC and MultiRC datasets for approaches that do not use external resources for training.
  • ARC: 56.82% F1 (41.24% on Challenge, 64.49% on Easy).
  • MultiRC: 26.1% EMO.
  • Significant QA Improvement: When coupled with a BERT-based QA model, ROCC improved performance by 9.15% accuracy on ARC and 8.3% EMO on MultiRC compared to baselines without justifications or using simpler retrieval methods.
  • Higher Quality Justifications: ROCC-selected justifications were qualitatively better than those from a strong IR baseline (BM25), leading to a 5.4% F1 improvement on MultiRC.
  • Domain Robustness: ROCC demonstrated over 10% better F1 score stability across domains in MultiRC compared to a supervised BERT-based sentence selection method.

Limitations and Future Work

  • Knowledge Aggregation Limits: The paper notes that performance can decrease for very large values of 'k' (number of sentences in a set), suggesting that knowledge aggregation is still an open research challenge.
  • Future Work: The authors envision using ROCC scores as a distant supervision signal to train supervised justification selection methods. They also suggest exploring semantic alignment methods for sentence matching, although their current lexical-based approach shows robustness.

Summary updated

Abstract

We propose an unsupervised strategy for the selection of justification sentences for multi-hop question answering (QA) that (a) maximizes the relevance of the selected sentences, (b) minimizes the overlap between the selected facts, and (c) maximizes the coverage of both question and answer. This unsupervised sentence selection method can be coupled with any supervised QA approach. We show that the sentences selected by our method improve the performance of a state-of-the-art supervised QA model on two multi-hop QA datasets: AI2's Reasoning Challenge (ARC) and Multi-Sentence Reading Comprehension (MultiRC). We obtain new state-of-the-art performance on both datasets among approaches that do not use external resources for training the QA system: 56.82% F1 on ARC (41.24% on Challenge and 64.49% on Easy) and 26.1% EM0 on MultiRC. Our justification sentences have higher quality than the justifications selected by a strong information retrieval baseline, e.g., by 5.4% F1 in MultiRC. We also show that our unsupervised selection of justification sentences is more stable across domains than a state-of-the-art supervised sentence selection method.
BibTeX
@inproceedings{yadav2019quick,
  author = {Vikas Yadav and Steven Bethard and Mihai Surdeanu},
  title = {{Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=GtOZse8xmcy},
}
Figure