SciTaiL: A Textual Entailment Dataset from Science Question Answering

Tushar Khot, Ashish Sabharwal, Peter ClarkAAAI 2018 · OpenReview

Abstract

We present a new dataset and model for textual entailment, derived from treating multiple-choice question-answering as an entailment problem. SciTail is the first entailment set that is created solely from natural sentences that already exist independently "in the wild" rather than sentences authored specifically for the entailment task. Different from existing entailment datasets, we create hypotheses from science questions and the corresponding answer candidates, and premises from relevant web sentences retrieved from a large corpus. These sentences are often linguistically challenging. This, combined with the high lexical similarity of premise and hypothesis for both entailed and non-entailed pairs, makes this new entailment task particularly difficult. The resulting challenge is evidenced by state-of-the-art textual entailment systems achieving mediocre performance on SciTail, especially in comparison to a simple majority class baseline. As a step forward, we demonstrate that one can improve accuracy on SciTail by 5% using a new neural model that exploits linguistic structure.

Introduction

Recognizing Textual Entailment (RTE) is a central task in natural language understanding, determining if a "premise" sentence supports a "hypothesis" sentence. While foundational, most existing datasets like SNLI or SICK are somewhat artificial, relying on crowd-workers to author hypotheses specifically for the dataset. This creates a gap: models trained on these datasets often struggle with "in the wild" text where premises and hypotheses are authored independently and exhibit higher linguistic complexity. SciTail addresses this by deriving entailment pairs from a practical goal: answering science questions using web-retrieved evidence.

The SciTail Dataset

The primary contribution of this work is the SciTail dataset, which consists of 27,026 examples (10,101 "entails" and 16,925 "neutral"). Hypotheses are generated from 4th and 8th-grade science questions and their correct answers, while premises are relevant sentences retrieved from a massive 280 GB web crawl.

Examples of science questions and answers transformed into entailment pairs within the SciTail dataset.

Because the premises and hypotheses were not written to match, SciTail avoids the linguistic biases common in crowd-sourced datasets. Furthermore, even "neutral" pairs in SciTail often exhibit high lexical overlap, preventing models from relying on simple word-matching heuristics and requiring a deeper understanding of semantic relationships.

Decomposed Graph Entailment Model (DGEM)

To tackle the unique challenges of SciTail, we introduce the Decomposed Graph Entailment Model (DGEM). Unlike standard models that treat sentences as simple sequences, DGEM leverages the inherent logical structure of science hypotheses.

Key components of the DGEM architecture include:

  1. Graph Construction: Using Open Information Extraction (Open IE) to convert the hypothesis into a graph of nodes (entities) and edges (relations).
  2. Node & Edge Attention: Computing fine-grained support for each individual node and edge in the hypothesis graph relative to the premise.
  3. Decomposition: By breaking the hypothesis down, the model can explicitly verify if the relationships described (the edges) are present in the premise, rather than just matching individual words.
The DGEM architecture decomposes the hypothesis into a graph to compute fine-grained support from the premise.

Results and Discussion

SciTail proved to be a significant challenge for existing state-of-the-art models. DGEM achieved the top score of 77.3% accuracy on the test set, outperforming the Decomposable Attention Model (72.3%) and ESIM (70.6%). Interestingly, a simple Ngram overlap baseline reached 70.6%, nearly matching complex neural models and highlighting the difficulty of moving beyond word-matching on this data.

Qualitative analysis shows that DGEM can correctly identify support for implicit relations, such as recognizing that "are formed" in a hypothesis is supported by the context of a premise even if the exact words are not identical. However, limitations remain; approximately 35% of DGEM's errors were attributed to incorrect Open IE extractions, and the model occasionally overlooks critical "bottleneck" words.

Conclusion

SciTail provides a more realistic and challenging benchmark for textual entailment by utilizing naturally occurring sentences from the science domain. Our proposed DGEM model demonstrates that incorporating linguistic structure through hypothesis decomposition can significantly improve performance on such complex, real-world data. Future research will focus on improving structural parsing and incorporating broader domain knowledge to handle more subtle reasoning tasks.

Summary updated

Abstract

We present a new dataset and model for textual entailment, derived from treating multiple-choice question-answering as an entailment problem. SciTail is the first entailment set that is created solely from natural sentences that already exist independently ``in the wild'' rather than sentences authored specifically for the entailment task. Different from existing entailment datasets, we create hypotheses from science questions and the corresponding answer candidates, and premises from relevant web sentences retrieved from a large corpus. These sentences are often linguistically challenging. This, combined with the high lexical similarity of premise and hypothesis for both entailed and non-entailed pairs, makes this new entailment task particularly difficult. The resulting challenge is evidenced by state-of-the-art textual entailment systems achieving mediocre performance on SciTail, especially in comparison to a simple majority class baseline. As a step forward, we demonstrate that one can improve accuracy on SciTail by 5% using a new neural model that exploits linguistic structure.
BibTeX
@inproceedings{khot2018scitail,
  author = {Tushar Khot and Ashish Sabharwal and Peter Clark},
  title = {{SciTaiL: A Textual Entailment Dataset from Science Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=Bk4SB1ZOWB},
}
Figure