Knowledge Enhanced Attention for Robust Natural Language Inference

Alexander Hanbo Li, Abhinav SethyarXiv 2019 · OpenReview

TL;DR

This paper tackles the fragility of Natural Language Inference (NLI) models when faced with adversarial examples that exploit lexical variations. They propose a novel method to enhance the attention mechanism by directly incorporating external knowledge (like synonyms or antonyms) into the model's layers, significantly improving robustness without requiring extra parameters. This approach achieves human-level performance on adversarial NLI benchmarks when combined with pre-trained models like BERT.

What Problem Does This Paper Address?

State-of-the-art neural network models for Natural Language Inference (NLI), a task that involves determining the relationship (entailment, contradiction, or neutral) between a premise and a hypothesis sentence, often perform exceptionally well on standard datasets. However, recent research has revealed a critical vulnerability: these models experience a significant drop in accuracy when presented with adversarial examples. These adversarial examples are specifically designed to trick the models by making small, often lexical, changes to sentences (e.g., replacing a word with its synonym or antonym) that humans would easily understand. This lack of robustness is a major concern for deploying NLI systems in real-world applications where such variations are common. The paper aims to bridge this gap by making NLI models more resilient to these lexical perturbations.

Methods

The core of the proposed solution lies in enhancing the attention mechanism, a fundamental component in modern neural networks, particularly Transformers. The authors propose a simple yet effective modification to the multi-head attention mechanism. Instead of solely relying on learned similarities between word embeddings, they introduce a way to directly inject external lexical knowledge.

Specifically, they leverage the concept of structured embeddings, where relationships between words (like synonymy, hypernymy, antonymy) are modeled. For each attention head, they propose to add an offset to the similarity score if two words exhibit a specific lexical relation. This offset is controlled by a hyperparameter b, which can be tuned. Essentially, if two words have a known lexical relationship (e.g., they are synonyms), their similarity score in the attention mechanism is boosted. This modification is applied directly within the attention calculation, making it a "quick fix" that doesn't require retraining the entire model from scratch or adding new parameters.

This method can be integrated into various NLI model architectures, including:

  1. Decomposable Models: These models typically use separate encoders for the premise and hypothesis, followed by a cross-encoder for interaction. The proposed knowledge enhancement is applied to the multi-head attention within the cross-encoder layers.
  2. Transformer Encoders (like BERT): For models that process concatenated premise and hypothesis, the knowledge is incorporated into the multi-head attention layers of the Transformer encoder.

Datasets

The paper uses the following datasets for training and evaluation:

  • Stanford NLI (SNLI): A widely used dataset for NLI tasks.
  • Multi-Genre NLI (MultiNLI): Another large-scale NLI dataset covering diverse genres.
  • Adversarial SNLI: A benchmark dataset specifically created to test the robustness of NLI models against lexical variations. This dataset is crucial for evaluating the effectiveness of the proposed method.

Key Insights and Ideas

  • Attention is the Bottleneck: The paper strongly suggests that the attention mechanism, while powerful, is a key area where NLI models fail to generalize under lexical variations. By directly injecting knowledge here, they can significantly improve robustness.
  • External Knowledge as a "Quick Fix": The proposed method is remarkably simple and doesn't require extensive architectural changes or additional trainable parameters. This makes it highly practical for improving existing models.
  • Pretraining + Knowledge = Superpowers: Combining their knowledge-enhanced attention with pre-trained models like BERT yields impressive results, suggesting that pretraining captures general language understanding, while their method adds specific robustness against lexical noise.
  • Layer-wise Impact: The paper investigates where to apply the knowledge enhancement. They find that applying it to earlier layers (especially the first cross-encoder layer) is often more effective, as it influences subsequent layers' learning.
  • Robustness vs. Accuracy on Clean Data: While the primary goal is robustness, the method also shows marginal improvements or minimal degradation on clean datasets, indicating it doesn't harm general performance.

Key Results

  • Significant Robustness Boost: The proposed method consistently improves accuracy on the adversarial SNLI dataset across different model architectures. For instance, a vanilla decomposable model (Model I) drops from 86.0% to 50.3% on adversarial data, but with the knowledge enhancement, it reaches 71.1% (a 20.8% absolute increase).
  • Human-Level Performance with BERT: When combined with BERT, the knowledge-enhanced model achieves an accuracy of 94.1% on the adversarial SNLI dataset, which is reported to be equal to estimated human performance.
  • Model Agnostic: The method is shown to be effective on both decomposable models and Transformer-based models like BERT, demonstrating its broad applicability.
  • Post-hoc Modification is Insufficient: Applying the external knowledge only during inference (post-hoc) does not yield significant improvements, highlighting the importance of incorporating knowledge during the training phase.

Limitations and Future Work

  • Focus on Lexical Knowledge: The current work primarily focuses on lexical knowledge (synonyms, antonyms, etc.). The authors acknowledge that other forms of external knowledge might also be beneficial.
  • Hyperparameter Tuning: The effectiveness of the method depends on the hyperparameter b, which controls the magnitude of the knowledge injection. Tuning this parameter is necessary.
  • Low Recall on Neutral Class: Despite improvements, the models still struggle with predicting neutral class examples on adversarial data, indicating that predicting nuanced relationships remains challenging.
  • Future Work: The authors suggest exploring other types of external knowledge and further investigating the impact of knowledge integration across different layers and model architectures. They also hint at the potential for this method to improve robustness in other NLP tasks beyond NLI.

Summary updated

Abstract

Neural network models have been very successful at achieving high accuracy on natural language inference (NLI) tasks. However, as demonstrated in recent literature, when tested on some simple adversarial examples, most of the models suffer a significant drop in performance. This raises the concern about the robustness of NLI models. In this paper, we propose to make NLI models robust by incorporating external knowledge to the attention mechanism using a simple transformation. We apply the new attention to two popular types of NLI models: one is Transformer encoder, and the other is a decomposable model, and show that our method can significantly improve their robustness. Moreover, when combined with BERT pretraining, our method achieves the human-level performance on the adversarial SNLI data set.
BibTeX
@misc{li2019knowledge,
  author = {Alexander Hanbo Li and Abhinav Sethy},
  title = {{Knowledge Enhanced Attention for Robust Natural Language Inference}},
  year = {2019},
  eprint = {1909.00102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.00102},
}
Figure