Introduction
Natural Language Inference (NLI) is a fundamental task in NLP, requiring a model to determine if a premise ($P$) entails a hypothesis ($H$). While modern Large Language Models (LLMs) like BERT achieve high accuracy on standard benchmarks, they often function as "black boxes." Research indicates these models frequently rely on "spurious cues"—statistical artifacts like word overlap or sentence length—rather than genuine semantic reasoning, a phenomenon known as the "Clever Hans" effect. This paper introduces AMR4NLI, a method that addresses these limitations by treating entailment as a semantic substructure problem, enhancing both interpretability and robustness.
Methodology
The core hypothesis of AMR4NLI is that if $P$ entails $H$, the semantic elements of $H$ should be contained within the structure of $P$. To operationalize this, the authors utilize Abstract Meaning Representation (AMR) graphs. AMR abstracts away from surface syntax to represent "who is doing what to whom" in a rooted, directed graph.
The framework employs asymmetric, precision-based metrics to measure this containment. Unlike symmetric similarity measures, these metrics specifically evaluate the extent to which the hypothesis structure is covered by the premise. Key metrics explored include:
- WWLKP: A contextualized graph matching metric.
- SmatchP: A metric based on structural triple matching.
- Hybrid Model: An ensemble combining fine-tuned BERT predictions with graph-based scores (WWLKP).
Experimental Evaluation
The approach was evaluated across five major English NLI benchmarks: SICK, SNLI (dev and test), and MNLI (matched and mismatched). Additionally, a "Hard" subset of SNLI was used to test robustness against known spurious biases.
Key Results and Discussion
High-Precision Semantic Signaling
A significant finding is that graph-based metrics serve as highly trustworthy predictors of entailment. When these metrics assign a high score, the probability of true entailment is exceptionally high. As shown in the table below, the WWLKP metric achieves an accuracy of 97.3% on the top 1% of most confident predictions, whereas BERT reaches only 84.5% in the same percentile.
Robustness to Spurious Cues
The structural approach demonstrates superior robustness compared to purely neural models. While fine-tuned BERT models suffer a "catastrophic" performance drop of 12.0 points in AUC when tested on "hard" datasets where statistical shortcuts are removed, AMR-based metrics remain stable or even show slight improvements. This suggests that graph-based methods rely on actual semantic content rather than dataset artifacts.
Hybrid Performance and Generalization
The hybrid model (BERT + WWLKP) achieved state-of-the-art results, with an average AUC of 84.8 compared to 80.3 for standalone BERT. The hybrid approach also showed better generalization on out-of-domain data (MNLI-mismatched), indicating that symbolic structures and neural models provide complementary information.
Conclusion
AMR4NLI demonstrates that explicit semantic structures can significantly improve the interpretability and robustness of NLI systems. While graph metrics currently face a "recall gap"—struggling when $P$ and $H$ use different structures for similar meanings—and are limited by the quality of automatic AMR parsers, they provide a high-precision signal that mitigates the weaknesses of black-box neural models. Future work will focus on incorporating better lexical semantics into the graph matching process to bridge the recall gap.
Summary updated