Transitive self-consistency evaluation of NLI models without gold labels

Wei Wu, Mark LastEMNLP 2025

TL;DR

This paper introduces a novel method for evaluating the transitive consistency of Natural Language Inference (NLI) models without relying on human-annotated "gold labels." By automatically generating adversarial examples through antonym replacement, they demonstrate that even state-of-the-art NLI models exhibit significant logical inconsistencies, suggesting a reliance on superficial cues rather than deep semantic understanding.

What Problem Does This Paper Address?

Natural Language Inference (NLI) models are designed to determine the logical relationship (entailment, contradiction, or neutral) between a premise and a hypothesis. While these models have shown impressive performance, recent research indicates they can be inconsistent in their decisions, even when presented with logically related statements. Existing methods for evaluating this "transitive consistency" often rely on expensive, human-annotated gold labels and only test a subset of the possible logical rules. This paper addresses the need for a more efficient and comprehensive evaluation of NLI model consistency, specifically focusing on whether models maintain logical coherence across a chain of inferences without external ground truth.

Methods

The core of this paper is a novel approach to generating adversarial examples for NLI models. The method involves:

  • Antonym Replacement: For sentences in existing NLI datasets, specific words (nouns, verbs, adjectives, adverbs) are identified. These words are then replaced with their antonyms using either:
  • WordNet: A lexical database that provides antonyms. The process involves filtering for antonyms with the same part-of-speech and selecting the one with the highest cosine similarity to the rest of the sentence.
  • LLMs: Large Language Models are prompted to replace a word with its antonym, with the assumption that LLMs handle grammaticality and semantic shift.
  • Linguistic Acceptability Filtering: To ensure the generated sentences are still meaningful, a BERT model fine-tuned on the Corpus of Linguistic Acceptability (CoLA) is used to filter out grammatically incorrect or nonsensical sentences.
  • Transitive Consistency Evaluation: The generated premise-hypothesis pairs, along with their modified hypotheses, are fed into NLI models. The models' own predicted labels for these pairs are then checked against four transitive inference rules (e.g., if P entails H, and H entails H', then P should entail H'). Inconsistencies arise when the model's labels violate these rules.

Datasets

The authors utilize existing benchmark NLI datasets for generating their adversarial examples:

  • SNLI (Stanford Natural Language Inference)
  • MNLI (Multi-Genre Natural Language Inference)
  • ANLI (Adversarial NLI)
  • WANLI (Worker and AI Collaboration for Natural Language Inference Dataset Creation)

The paper does not introduce a new benchmark dataset but rather a methodology to create adversarial test sets from existing ones.

Key Insights and Ideas

  • Self-Consistency is Crucial: The paper emphasizes that for reliable AI systems, internal self-consistency is paramount, independent of the absolute correctness of labels against ground truth.
  • Antonym Replacement as a Powerful Tool: Automatically generating antonym-based adversarial examples is an effective way to probe for logical inconsistencies without the need for expensive human annotation.
  • LLMs vs. WordNet for Antonyms: While LLMs can generate more natural-sounding sentences and handle grammaticality, their antonym selections are not necessarily superior to WordNet, and they are computationally more expensive.
  • NLI Models Rely on Superficial Cues: The observed inconsistencies suggest that NLI models often rely on surface-level lexical cues rather than a deep understanding of semantic relationships, making them vulnerable to adversarial attacks.
  • Transitive Rules Vary in Difficulty: Models struggle more with maintaining consistency for entailment-based transitive rules (rules 1 and 2) compared to those involving neutral cases (rules 3 and 4).

Key Results

  • Widespread Inconsistency: The experiments revealed that all evaluated state-of-the-art NLI models exhibit significant transitive inconsistencies.
  • High Rate of Contradictions: For the generated adversarial examples, a majority (around 60%) were labeled as mutual contradictions by the models, as expected from antonym replacements.
  • Dataset Performance Variation: Models performed better on SNLI and MNLI but struggled more with ANLI, which contains human-generated adversarial examples. WANLI showed intermediate performance.
  • Acceptability of Modified Sentences: The WordNet-based antonym replacement algorithm achieved an acceptability rate of around 75-87% across datasets after filtering, indicating the generated examples are largely coherent.
  • Inconsistency Rates: Table 4 shows inconsistency rates for different transitive rules, with violations of E&E → !E and E&C → !C being more frequent. For instance, the bart-large model showed a 29.15% inconsistency rate for E&C → !C on MNLI.

Limitations and Future Work

  • Antonym Replacement Limitations: Some antonym replacements can lead to entailments rather than contradictions, and WordNet's coverage of antonyms is limited. The paper suggests exploring other sources for antonyms and enhancing the filtering pipeline.
  • Focus on Transitivity: The current work focuses solely on transitive consistency. Future work could extend this to other logical consistencies like negational and symmetrical consistency.
  • LLM Evaluation: The framework could be applied to evaluate the logical consistency of other AI systems like ChatGPT and Gemini.
  • English-Centric: Current experiments are limited to English-based models and datasets.
  • Fixed Phrases and Polysemy: The paper notes that fixed phrases and polysemous words can still lead to acceptable but nonsensical sentences that are difficult to filter automatically.

Summary updated

BibTeX
@inproceedings{wu2025transitive,
  author = {Wei Wu and Mark Last},
  title = {{Transitive self-consistency evaluation of NLI models without gold labels}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure