Abstract
We propose a simple neural architecture for natural language inference (NLI) that utilizes attention to decompose the problem into parallelizable subproblems. By focusing on local alignments rather than complex sequential representations, our model achieves state-of-the-art results on the Stanford Natural Language Inference (SNLI) dataset. Notably, the approach uses nearly an order of magnitude fewer parameters than contemporary LSTM-based models and does not rely on word-order information in its vanilla form.
Introduction
Natural Language Inference (NLI) involves determining the logical relationship between a premise and a hypothesis: entailment, contradiction, or neutrality. At the time of this research, the dominant trend in NLP was to build increasingly deep and complex representations using Long Short-Term Memory (LSTM) networks or Convolutional Neural Networks (CNNs). While effective, these models are computationally expensive and difficult to parallelize due to their sequential nature.
This paper challenges the assumption that NLI requires such complex, deep representations. The authors demonstrate that a simpler "Attend-Compare-Aggregate" framework can be more efficient and accurate by focusing on aligning local bits of text.
The Attend-Compare-Aggregate Framework
The proposed architecture is "fully decomposable," meaning computations for different words can be performed independently and in parallel. The model consists of three main stages:
- Attend: The model creates a "soft alignment" between words in the premise and the hypothesis. It uses a simple feed-forward network to compute attention weights between every pair of words, reducing complexity by applying the network to each word individually before computing dot products.
- Compare: For each word, a comparison vector is created by looking at the word itself and its aligned counterpart from the other sentence, processed via another feed-forward network.
- Aggregate: The model sums the comparison vectors for both sentences and passes the result through a final classifier to predict the logical relationship.
While the vanilla model ignores word order, the authors also introduced an optional Intra-Sentence Attention step. This allows the model to capture compositional relationships (e.g., "not sleep") within a single sentence before the alignment phase.
Experimental Results
The model was evaluated on the Stanford Natural Language Inference (SNLI) dataset, which contains approximately 570,000 human-written sentence pairs.
Performance and Efficiency
The results demonstrate that the decomposable attention model is both highly accurate and remarkably efficient. The vanilla model achieved 86.3% accuracy, while the version with intra-sentence attention reached 86.8%.
As shown in the table below, the proposed approach outperforms much larger models. For instance, it achieves better accuracy than the 450D LSTMN while using only 382K to 582K parameters, compared to the millions of parameters required by LSTM and GRU-based encoders.
Key Insights
- Alignment Over Representation: Aligning specific words (e.g., "Bob" with "Bob") is often more critical for NLI than building a holistic, high-dimensional vector of the entire sentence.
- Parallelism: Relying on feed-forward networks instead of LSTMs makes the model "trivially parallelizable," significantly speeding up training and deployment.
- Order Independence: Surprisingly, the vanilla model—which treats sentences as "bags of words" during alignment—outperformed many models that strictly respected word order.
Conclusion
The decomposable attention model proves that for NLI, a "less is more" approach can be highly effective. By prioritizing attention-based alignment and parallelizable feed-forward layers, the model achieves state-of-the-art performance with a fraction of the parameters and computational cost of previous deep learning architectures. While it may struggle with rare cases where word order is the sole determinant of meaning, its efficiency and accuracy make it a powerful baseline for natural language understanding tasks.
Summary updated