TL;DR
MENLI introduces a new paradigm for evaluating AI-generated text by using Natural Language Inference (NLI) instead of just semantic similarity. By focusing on logical entailment, MENLI is significantly more robust against adversarial attacks—such as swapped names or numbers—and, when combined with existing metrics, boosts both accuracy and reliability across machine translation and summarization tasks.
The Blind Spot of Semantic Similarity
For years, the field of Natural Language Processing (NLP) relied on simple word-overlap counts like BLEU or ROUGE. Recently, there has been a shift toward BERT-based metrics (e.g., BERTScore) that measure semantic similarity. However, these metrics possess a dangerous "blind spot": similarity is not synonymous with correctness.
As the authors demonstrate, current metrics are easily fooled by "adversarial" changes. A sentence like "5 soldiers were wounded" is semantically very similar to "50,000 soldiers were killed" in terms of context and actors, but the latter represents a catastrophic failure in translation or summarization. Because the overall "vibe" of the sentence remains similar, similarity-based metrics often assign high scores to factually incorrect outputs.
Methodology: The MENLI Framework
The authors propose MENLI (MEtrics from NLI) to address this gap. Instead of asking "How similar are these two sentences?", MENLI asks "Does sentence A logically entail sentence B?"
- The NLI Paradigm: The framework leverages pre-trained NLI models that categorize the relationship between two sentences as Entailment, Contradiction, or Neutral.
- Pooling and Directionality: The researchers experimented with 15 different strategies to convert these probabilities into a single score. For translation, they utilized bi-implication (A implies B AND B implies A), while for summarization, they verified if the source document logically entails the summary.
- Ensembling: Recognizing that NLI is optimized for logic but may overlook surface-level fluency (like typos), the authors created combined metrics. By adding a small weight of MENLI to standard metrics like BERTScore or COMET, they achieved a "best of both worlds" evaluator that is both fluent and logically sound.
The "Logic Guard": Robustness Against Adversarial Attacks
NLI models act as a logical filter. While a model like BERTScore might give a high score to a sentence with a wrong name because the rest of the sentence matches, the NLI model flags it as a "Contradiction," significantly lowering the final score. This is particularly effective against common errors that similarity metrics miss, such as pronoun shifts, name mismatches, and verb negations.
Key Results and Insights
The researchers evaluated MENLI on massive standard benchmarks (WMT15-21, RealSumm, SummEval) and specialized adversarial datasets (PAWS, SEadv). Their findings highlight three major advantages:
- Superior Robustness: In reference-based translation, NLI-based metrics achieved >85% accuracy in identifying errors, whereas standard metrics often hovered between 32% and 78%.
- The Concave Performance Boost: Adding just a 20-30% weight of MENLI to existing state-of-the-art metrics results in a "concave" performance boost. This improves correlation with human judgments on standard benchmarks by 5% to 6% while simultaneously increasing adversarial robustness by 15% to 30%.
- Summarization Factuality: In summarization tasks, MENLI (specifically versions based on DeBERTa) outperformed almost all existing metrics in measuring factual consistency and fluency.
Conclusion and Future Directions
The authors argue that trying to "fix" similarity-based metrics with better token matching is a dead end for factuality. To catch factual errors, a model must understand the logical relationship between the premise and the hypothesis. While MENLI currently faces challenges with long-form document granularity and the "fluency gap" (where it may miss minor typos), it represents a significant step toward more reliable and logically aware AI evaluation.
Summary updated