A Decomposable Attention Model for Natural Language Inference

Ankur P. Parikh, Oscar Täckström, Dipanjan Das, Jakob Uszkoreit

Abstract

We propose a simple neural architecture for natural language inference. Our approach uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable. On the Stanford Natural Language Inference (SNLI) dataset, we obtain state-of-the-art results with almost an order of magnitude fewer parameters than previous work and without relying on any word-order information. Adding intra-sentence attention that takes a minimum amount of order into account yields further improvements.

Introduction

Natural Language Inference (NLI) is the task of determining the logical relationship between a premise and a hypothesis. The goal is to classify the relationship as entailment, contradiction, or neutral.

In 2016, the dominant trend in NLP was the use of complex, sequential models like LSTMs or CNNs to create global sentence representations. However, these models are computationally expensive. This paper argues that NLI can often be solved by aligning local text segments—such as recognizing that "Bob is awake" follows from "Bob cannot sleep"—rather than requiring a full global syntactic analysis.

Methodology

The proposed "Decomposable Attention Model" consists of three main stages that are fully parallelizable:

  1. Attend: The model computes a "soft alignment" between every word in the premise and every word in the hypothesis using a feed-forward network to determine relevance.
  2. Compare: For each word, a comparison vector is created by looking at the word alongside its aligned counterpart from the other sentence. Each word-pair comparison is handled independently.
  3. Aggregate: The comparison vectors are summed for each sentence and passed through a final feed-forward classifier.
Overview of the Decomposable Attention Model architecture showing the Attend, Compare, and Aggregate stages.

The "vanilla" version of this model ignores word order entirely, treating sentences as bags of words. An optional Intra-Sentence Attention module can be added to capture local sequence information before the alignment step.

Experimental Setup

The model was evaluated on the Stanford Natural Language Inference (SNLI) dataset, which contains approximately 570,000 sentence pairs. At the time, this was the largest and most challenging benchmark for NLI, requiring models to demonstrate true semantic understanding beyond simple lexical overlap.

Results and Discussion

The Decomposable Attention Model achieved state-of-the-art performance while maintaining extreme parameter efficiency.

Performance comparison on the SNLI dataset, highlighting the proposed model's accuracy and low parameter count.

Key Findings:

  • State-of-the-Art Accuracy: The vanilla model achieved 86.3% accuracy, while the version with intra-sentence attention reached 86.8%, outperforming much larger models like the 450D LSTMN.
  • Parameter Efficiency: The model uses only ~380k to 580k parameters, nearly 10x fewer than competing LSTM-based models (which used 3M to 15M parameters).
  • Parallelism and Speed: By avoiding recurrent connections, the model can process all words simultaneously, making it significantly faster for training and inference.
  • The Role of Word Order: Surprisingly, a model with zero knowledge of word order outperformed complex sequential models, suggesting that local semantic alignment is often more critical than global syntactic structure for NLI.

Limitations and Future Work

While highly effective, the model has limitations:

  • Sequence Logic: Word order is critical for certain inferences (e.g., "The dog bit the man" vs. "The man bit the dog").
  • Fine-Grained Semantics: The model may struggle when a single word change (e.g., "open" vs. "closed") flips the entire meaning.
  • Numerical Reasoning: Challenges remain in handling examples requiring specific quantitative inference.

Future work could involve incorporating more sophisticated word representations or minimal structural information to address cases where word order is indispensable.

Summary updated

Abstract

We propose a simple neural architecture for natural language inference. Our approach uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable. On the Stanford Natural Language Inference (SNLI) dataset, we obtain state-of-the-art results with almost an order of magnitude fewer parameters than previous work and without relying on any word-order information. Adding intra-sentence attention that takes a minimum amount of order into account yields further improvements.
BibTeX
@inproceedings{parikh2016decomposable,
  author = {Ankur P. Parikh and Oscar Täckström and Dipanjan Das and Jakob Uszkoreit},
  title = {{A Decomposable Attention Model for Natural Language Inference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  url = {https://openreview.net/forum?id=H1NCabf_bS},
}
Figure