Introduction
Natural Language Inference (NLI) is often considered the "litmus test" for true language understanding. It requires a model to determine if a hypothesis is an entailment, contradiction, or neutral relative to a given premise. While modern neural networks achieve near-human performance on standard benchmarks like SNLI and MNLI, this research investigates whether these models are truly learning the compositional structure of language or merely exploiting word-level statistical shortcuts.
Methodology
The authors propose that current models are "compositionality-blind," relying on a "bag-of-words" approach rather than syntax. To prove this, they developed a three-pronged investigative framework:
1. Rule-Based Adversarial Sets
The researchers created two targeted datasets to test sensitivity to word order and modification:
- SOSWAP: Swapping the subject and object (e.g., "A woman is pulling a child" vs. "A child is pulling a woman").
- ADDAMOD: Moving an adjective modifier to a different noun (e.g., "A yellow cat sits in grass" vs. "A cat sits in yellow grass").
2. Lexically-Misleading Score (LMS)
The paper's primary contribution is the Lexically-Misleading Score (LMS). By training a simple Bag-of-Words (BoW) model, the authors identified examples where lexical cues (like the presence of the word "not") strongly suggest a label that is actually incorrect. These "hard" examples form the Compositionality-Sensitivity (CS) subsets, which require structural understanding to solve.
3. Ablation and Shuffling
The authors tested model resilience by training on shuffled input words. They found that shuffling only caused a 2-3% drop in performance on standard benchmarks, confirming that these datasets can be largely solved without considering word order.
Key Results
- The Compositionality Gap: On the $CS_{0.7}$ subset—where lexical clues are intentionally misleading—human performance remained high (80-86%), but state-of-the-art model performance plummeted to 50-57%.
- Total Failure on Swaps: On the SOSWAP test, models that scored 88% on standard SNLI dropped to nearly 0% accuracy on the correct label. Because the word overlap was 100%, models biased toward lexical similarity incorrectly classified contradictions as entailments.
- Architecture Matters: While most models perform similarly on standard tests, the CS evaluation revealed that tree-based models (like S-TLSTM) handle structural changes significantly better than simple sequential RNNs.
Conclusion
The study concludes that current NLI benchmarks are "lexically easy" and do not sufficiently penalize models for ignoring compositionality. The authors call for the development of "lexically-balanced" datasets that force models to move beyond keyword matching and toward a genuine understanding of linguistic structure.
Summary updated