AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples

Dongyeop Kang, Tushar Khot, Ashish Sabharwal, Eduard H. Hovy

Abstract

We consider the problem of learning textual entailment models with limited supervision (5K-10K training examples), and present two complementary approaches for it. First, we propose knowledge-guided adversarial example generators for incorporating large lexical resources in entailment models via only a handful of rule templates. Second, to make the entailment model - a discriminator - more robust, we propose the first GAN-style approach for training it using a natural language example generator that iteratively adjusts based on the discriminator's performance. We demonstrate effectiveness using two entailment datasets, where the proposed methods increase accuracy by 4.7% on SciTail and by 2.8% on a 1% training sub-sample of SNLI. Notably, even a single hand-written rule, negate, improves the accuracy on the negation examples in SNLI by 6.1%.

Introduction

Textual entailment, also known as Natural Language Inference (NLI), is the task of determining whether a premise sentence entails, contradicts, or is neutral toward a hypothesis. While deep learning models have achieved near-human performance on large-scale benchmarks like SNLI, they are often brittle and prone to failure when faced with simple linguistic variations. Models frequently overfit to annotation artifacts in crowd-sourced data and lack common-sense lexical knowledge, such as understanding that "dog" entails "animal."

Integrating external knowledge bases like WordNet into neural architectures is traditionally difficult, and simple techniques like retrofitting word embeddings often yield minimal gains. This paper introduces AdvEntuRe (Adversarial Entailment using Resources), a framework that uses knowledge-guided rules and neural generators to create challenging training examples on the fly, significantly boosting model robustness in low-resource settings.

The AdvEntuRe Framework

AdvEntuRe employs a Generative Adversarial Network (GAN) style approach. Instead of generating realistic data, the goal is to train a highly robust discriminator (the entailment model itself) by forcing it to compete against generators that produce increasingly difficult adversarial examples.

The AdvEntuRe framework: A discriminator D is trained against a neural generator and rule-based generators.

Adversarial Example Generation

The framework utilizes three types of generators to expose the discriminator's weaknesses:

  1. Knowledge-Guided Generators ($\mathbb{G}^{\text{KB}}$): These use simple rule templates to swap words based on external databases. Resources include WordNet (for synonyms, hypernyms, and antonyms), PPDB (for paraphrases), and SICK (for compositional rules).
  2. Hand-Defined Generators ($\mathbb{G}^{\text{H}}$): Procedural rules designed for specific linguistic phenomena. For instance, a negation rule adds "not" or "did not" to main verbs to explicitly teach the model how negation flips entailment to contradiction.
  3. Neural Generators ($\mathbb{G}^{\text{s2s}}$): A sequence-to-sequence model trained to generate a new hypothesis from a given premise for a specific entailment class. This generator is updated iteratively using the discriminator's loss as a reward signal.
Summary of knowledge sources and rule templates used for generating adversarial examples.

Experimental Evaluation

The models were evaluated on the SNLI and SciTail datasets. To test the framework's effectiveness in low-resource scenarios, the authors evaluated performance on sub-sampled versions of these datasets (1%, 10%, 50%, and 100% of training data).

Key Results

  • Significant Low-Data Gains: On a 1% sub-sample of SNLI (~5K examples), AdvEntuRe improved accuracy by 2.8% over the baseline. On a 1% sub-sample of SciTail, it improved accuracy by a massive 9.18%.
  • State-of-the-Art on SciTail: Using the full SciTail dataset, the rule-augmented model achieved 79.03% accuracy, outperforming the baseline by 4.7% and setting a new state-of-the-art.
  • Negation Mastery: By applying a single hand-written negation rule, the model's accuracy on negation-heavy SNLI test examples jumped by 6.1% (from 76.64% to 82.74%).

Conclusion

AdvEntuRe demonstrates that task-specific data augmentation is more effective than task-agnostic embedding adjustments for injecting knowledge into NLI models. The framework proves that even a handful of well-crafted rules can fix glaring logical blind spots in deep learning models. While neural generators are highly effective for regularizing models in low-data regimes, rule-based generators provide consistent value even as the training set size increases.

Summary updated

Abstract

We consider the problem of learning textual entailment models with limited supervision (5K-10K training examples), and present two complementary approaches for it. First, we propose knowledge-guided adversarial example generators for incorporating large lexical resources in entailment models via only a handful of rule templates. Second, to make the entailment model - a discriminator - more robust, we propose the first GAN-style approach for training it using a natural language example generator that iteratively adjusts based on the discriminator's performance. We demonstrate effectiveness using two entailment datasets, where the proposed methods increase accuracy by 4.7% on SciTail and by 2.8% on a 1% training sub-sample of SNLI. Notably, even a single hand-written rule, negate, improves the accuracy on the negation examples in SNLI by 6.1%.
BibTeX
@inproceedings{kang2018adventure,
  author = {Dongyeop Kang and Tushar Khot and Ashish Sabharwal and Eduard H. Hovy},
  title = {{AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=ry-_0ogO-H},
}
Figure