AMR4NLI: Interpretable and robust NLI measures from semantic graphs

Juri Opitz, Shira Wein, Julius Steen, Anette Frank, Nathan SchneiderarXiv 2023 · DOI · arXiv

Introduction

Natural Language Inference (NLI) is a fundamental task in NLP, requiring a model to determine if a premise ($P$) entails a hypothesis ($H$). While modern Large Language Models (LLMs) like BERT achieve high accuracy on standard benchmarks, they often function as "black boxes." Research indicates these models frequently rely on "spurious cues"—statistical artifacts like word overlap or sentence length—rather than genuine semantic reasoning, a phenomenon known as the "Clever Hans" effect. This paper introduces AMR4NLI, a method that addresses these limitations by treating entailment as a semantic substructure problem, enhancing both interpretability and robustness.

Methodology

The core hypothesis of AMR4NLI is that if $P$ entails $H$, the semantic elements of $H$ should be contained within the structure of $P$. To operationalize this, the authors utilize Abstract Meaning Representation (AMR) graphs. AMR abstracts away from surface syntax to represent "who is doing what to whom" in a rooted, directed graph.

The framework employs asymmetric, precision-based metrics to measure this containment. Unlike symmetric similarity measures, these metrics specifically evaluate the extent to which the hypothesis structure is covered by the premise. Key metrics explored include:

  • WWLKP: A contextualized graph matching metric.
  • SmatchP: A metric based on structural triple matching.
  • Hybrid Model: An ensemble combining fine-tuned BERT predictions with graph-based scores (WWLKP).

Experimental Evaluation

The approach was evaluated across five major English NLI benchmarks: SICK, SNLI (dev and test), and MNLI (matched and mismatched). Additionally, a "Hard" subset of SNLI was used to test robustness against known spurious biases.

Key Results and Discussion

High-Precision Semantic Signaling

A significant finding is that graph-based metrics serve as highly trustworthy predictors of entailment. When these metrics assign a high score, the probability of true entailment is exceptionally high. As shown in the table below, the WWLKP metric achieves an accuracy of 97.3% on the top 1% of most confident predictions, whereas BERT reaches only 84.5% in the same percentile.

Accuracy scores at top-k percentiles demonstrating the high precision of graph-based metrics.

Robustness to Spurious Cues

The structural approach demonstrates superior robustness compared to purely neural models. While fine-tuned BERT models suffer a "catastrophic" performance drop of 12.0 points in AUC when tested on "hard" datasets where statistical shortcuts are removed, AMR-based metrics remain stable or even show slight improvements. This suggests that graph-based methods rely on actual semantic content rather than dataset artifacts.

AUC delta on hard datasets, highlighting the vulnerability of BERT compared to the stability of AMR-based metrics.

Hybrid Performance and Generalization

The hybrid model (BERT + WWLKP) achieved state-of-the-art results, with an average AUC of 84.8 compared to 80.3 for standalone BERT. The hybrid approach also showed better generalization on out-of-domain data (MNLI-mismatched), indicating that symbolic structures and neural models provide complementary information.

Conclusion

AMR4NLI demonstrates that explicit semantic structures can significantly improve the interpretability and robustness of NLI systems. While graph metrics currently face a "recall gap"—struggling when $P$ and $H$ use different structures for similar meanings—and are limited by the quality of automatic AMR parsers, they provide a high-precision signal that mitigates the weaknesses of black-box neural models. Future work will focus on incorporating better lexical semantics into the graph matching process to bridge the recall gap.

Summary updated

Abstract

The task of natural language inference (NLI) asks whether a given premise (expressed in NL) entails a given NL hypothesis. NLI benchmarks contain human ratings of entailment, but the meaning relationships driving these ratings are not formalized. Can the underlying sentence pair relationships be made more explicit in an interpretable yet robust fashion? We compare semantic structures to represent premise and hypothesis, including sets of contextualized embeddings and semantic graphs (Abstract Meaning Representations), and measure whether the hypothesis is a semantic substructure of the premise, utilizing interpretable metrics. Our evaluation on three English benchmarks finds value in both contextualized embeddings and semantic graphs; moreover, they provide complementary signals, and can be leveraged together in a hybrid model.
BibTeX
@misc{opitz2023amr,
  author = {Juri Opitz and Shira Wein and Julius Steen and Anette Frank and Nathan Schneider},
  title = {{AMR4NLI: Interpretable and robust NLI measures from semantic graphs}},
  year = {2023},
  eprint = {2306.00936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.00936},
}
Figure