Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering

Vikas Yadav, Steven Bethard, Mihai Surdeanu

Abstract

Vikas Yadav, Steven Bethard, Mihai Surdeanu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

In multi-hop Question Answering (QA), models must synthesize information across multiple sentences or documents to arrive at a correct answer. For instance, answering "What organ system does the esophagus and the liver belong to?" requires connecting disparate facts about anatomy. To ensure these models are interpretable and trustworthy, it is essential to provide justifications: the specific set of sentences that constitute the evidence for an answer.

Current approaches to justification selection typically fall into two categories: Information Retrieval (IR) baselines and supervised learning models. IR methods like BM25 often retrieve redundant or incomplete information. Supervised models, while more accurate, require expensive human-annotated "gold" justifications and often fail to generalize across different domains. This paper introduces ROCC, an unsupervised method that selects high-quality, diverse, and complete justification sets without the need for labeled training data.

The ROCC Framework

The authors propose ROCC (Relevance, Overlap, Coverage), an algorithm designed to evaluate sets of sentences rather than individual ones. The pipeline consists of four primary stages:

  1. Candidate Retrieval: A standard IR tool (BM25) retrieves the top 20 candidate sentences.
  2. Subset Generation: All possible combinations (subsets) of these sentences are generated.
  3. ROCC Scoring: Subsets are ranked based on a formula balancing three factors:
  4. Relevance (R): The semantic match between the sentences, the question, and the candidate answer.
  5. Overlap (O): A penalty for redundant information shared between sentences, ensuring each piece of evidence adds unique value.
  6. Coverage (C): A measure of whether the set collectively mentions all key terms in the query.
  7. Classification: The highest-scoring set is passed to a BERT-based classifier to validate the candidate answer.
Overview of the ROCC pipeline for multi-hop question answering.

By treating justification as a set-covering problem, ROCC ensures that the selected evidence is complementary rather than merely repetitive.

Experimental Setup and Datasets

ROCC was evaluated on two challenging multi-hop benchmarks:

  • MultiRC (Multi-Sentence Reading Comprehension): Requires reasoning across multiple sentences within a single paragraph.
  • ARC (AI2 Reasoning Challenge): Contains difficult grade-school science questions requiring reasoning over a knowledge base of 14 million sentences.

Results and Discussion

ROCC achieved state-of-the-art performance among models that do not utilize external training resources. On the ARC dataset, ROCC reached an F1 score of 56.82%, outperforming the BM25 baseline and improving overall QA accuracy by 9.15% compared to models using no justifications.

Performance comparison on the ARC dataset, showing ROCC's improvements over baselines.

Key findings include:

  • Robustness: As an unsupervised method, ROCC is more stable across different domains (e.g., science vs. fiction) than supervised BERT-based selectors, which often overfit to dataset-specific biases.
  • Transfer Learning: Pre-training the answer classifier on all retrieved sentences before fine-tuning on ROCC-selected justifications provided a significant performance boost.
  • Quality: Manual evaluations confirmed that ROCC-selected justifications are more complete and less redundant than those from standard IR methods.

Conclusion

ROCC demonstrates that unsupervised selection of justification sets can match or exceed the performance of complex supervised systems while offering greater domain robustness. Future work involves addressing the exponential growth of sentence combinations as the number of "hops" increases and using ROCC scores as distant supervision to train faster, supervised models.

Summary updated

BibTeX
@inproceedings{yadav2019quick,
  author = {Vikas Yadav and Steven Bethard and Mihai Surdeanu},
  title = {{Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=GtOZse8xmcy},
}
Figure