Abstract
We consider the problem of learning textual entailment models with limited supervision (5K-10K training examples), and present two complementary approaches for it. First, we propose knowledge-guided adversarial example generators for incorporating large lexical resources in entailment models via only a handful of rule templates. Second, to make the entailment model - a discriminator - more robust, we propose the first GAN-style approach for training it using a natural language example generator that iteratively adjusts based on the discriminator's performance. We demonstrate effectiveness using two entailment datasets, where the proposed methods increase accuracy by 4.7% on SciTail and by 2.8% on a 1% training sub-sample of SNLI. Notably, even a single hand-written rule, negate, improves the accuracy on the negation examples in SNLI by 6.1%.
Introduction
Textual Entailment (TE) is a fundamental task in natural language processing that requires determining if a premise sentence ($P$) logically entails a hypothesis sentence ($H$). While modern deep learning models achieve high performance on large-scale datasets like SNLI, they often rely on "annotation artifacts"—spurious correlations or lexical shortcuts—rather than genuine logical reasoning. This reliance makes models brittle; they may fail on simple linguistic variations, such as word reordering or negation, that were not prevalent in the training distribution. The AdvEntuRe framework addresses this by augmenting training with adversarial examples generated from external knowledge and adaptive neural models.
The AdvEntuRe Framework
The AdvEntuRe (Adversarial training for Entailment using Resource-based examples) framework consists of an entailment model acting as a discriminator ($D$) and a suite of generators ($G$) that produce challenging training pairs.
The framework employs three distinct types of generators to cover various linguistic phenomena:
- Knowledge-Guided Generators ($G^{KB}$): These leverage structured lexical resources like WordNet and PPDB to perform word-level substitutions (e.g., replacing a word with its hypernym or synonym).
- Hand-Authored Rules ($G^H$): These are high-leverage templates designed to target specific weaknesses, such as the
NEGATErule which transforms a sentence into its logical opposite. - Neural Generators ($G^{s2s}$): A sequence-to-sequence model trained to generate a hypothesis given a premise and a target entailment label.
Adversarial Training and Second-Order Generation
A key innovation of AdvEntuRe is its GAN-style training loop. The neural generator ($G^{s2s}$) is not static; it is fine-tuned using the discriminator's loss as a reward signal. This encourages the generator to discover "blind spots" in the discriminator, producing examples that the current model finds difficult to classify correctly.
Furthermore, the framework utilizes Second-Order Generation to expand the variety of training data. By applying a transformation rule to an existing hypothesis $H$ to create $H'$, the system uses logical composition tables to infer the relationship between the original premise $P$ and the new hypothesis $H'$. This allows the model to learn complex transitive relationships without manual labeling.
Experimental Results
The authors evaluated AdvEntuRe on the SciTail dataset and sub-samples of the SNLI dataset to test performance in low-to-moderate data regimes.
- SciTail: The framework achieved a 4.7% accuracy increase, setting a new state-of-the-art for the dataset.
- SNLI (1% sample): Accuracy improved by 2.8%, demonstrating the value of knowledge-guided augmentation when labeled data is scarce.
- Negation Robustness: The inclusion of a single
NEGATErule led to a 6.1% improvement on negation-specific test cases in SNLI, highlighting the effectiveness of targeted rule-based augmentation.
Conclusion
AdvEntuRe demonstrates that injecting external knowledge via adversarial data augmentation is a powerful, model-agnostic alternative to architectural modifications. By combining structured lexical resources with adaptive neural generation, the framework significantly enhances the robustness of textual entailment models against common linguistic pitfalls and data artifacts. Future work may focus on automating the discovery of complex linguistic rules to further scale the diversity of generated adversarial examples.
Summary updated