Evaluating Paraphrastic Robustness in Textual Entailment Models

Dhruv Verma, Yash Kumar Lal, Shreyashee Sinha, Benjamin Van Durme, Adam Poliak

Introduction

Recognizing Textual Entailment (RTE) is a foundational task in Natural Language Understanding (NLU), requiring models to determine if a premise logically entails a hypothesis. A truly robust model should maintain its prediction regardless of how the input is phrased, provided the core meaning remains unchanged. This property, known as paraphrastic robustness, is often overlooked in standard evaluations where models may rely on "surface cues"—statistical shortcuts like word overlap—rather than deep logic. This paper introduces $\hat{PaRTE}$ (Paraphrastic Robustness in Textual Entailment), a new benchmark designed to measure model consistency under linguistic variation.

Methodology

The researchers developed a rigorous multi-stage pipeline to construct the $\hat{PaRTE}$ dataset, ensuring high quality and semantic consistency:

  1. Generation: Using a T5 model fine-tuned on the PAWS dataset, the researchers generated multiple paraphrases for premises and hypotheses from the Pascal RTE1-3 challenges.
  2. Diversity Filtering: To ensure significant linguistic variation, a Jaccard index threshold of 0.75 was applied, filtering out paraphrases with excessive lexical overlap with the original text.
  3. Human Verification: Amazon Mechanical Turk workers verified the grammaticality and semantic equivalence of the generated pairs.
  4. Expert Audit: A final manual check identified "semantic variability"—subtle changes in wording that might alter the logical relationship (e.g., changing "expected to" to "will"). Approximately 4% of examples were discarded during this phase to maintain ground-truth integrity.

The $\hat{PaRTE}$ Benchmark

The resulting $\hat{PaRTE}$ dataset consists of 1,126 human-verified pairs. It includes 402 examples with a paraphrased premise, 602 with a paraphrased hypothesis, and 122 where both were rewritten. The study evaluated five architectures: a simple Bag-of-Words (BoW), a BiLSTM, BERT, RoBERTa, and GPT-3 (evaluated in a zero-shot setting).

Key Insights

Accuracy vs. Robustness

Traditional accuracy metrics can be deceptive. A model might maintain the same overall accuracy score while "flipping" its predictions on individual examples when they are paraphrased. The authors argue that the percentage of changed predictions ($\% \Delta \hat{PaRTE}$) is the true measure of robustness.

Fragility on Incorrect Predictions

A significant finding is that models are far more likely to change their predictions on examples they initially failed to classify correctly. For instance, RoBERTa changed its prediction on only 5.5% of examples it originally got right, but flipped on nearly 21% of those it originally got wrong. This suggests that when a model lacks a clear logical signal, paraphrasing acts as a randomizing force.

Model prediction changes based on initial correctness (Blue: Correct, Red: Incorrect)

The Double Paraphrase Penalty

Robustness decreases further when both the premise and hypothesis are paraphrased. For RoBERTa, the flip rate increased from approximately 7% for single-sentence changes to 12% for double-sentence changes.

Results

The evaluation reveals that even state-of-the-art models exhibit significant fragility. While RoBERTa emerged as the most robust model, it still failed to remain consistent on roughly 1 out of every 12 paraphrased examples. Notably, GPT-3 showed less robustness than BERT despite its higher general performance, highlighting that performance does not always equate to linguistic stability.

Comparison of model performance and robustness across datasets

Conclusion

The $\hat{PaRTE}$ benchmark highlights a critical gap in the linguistic robustness of modern NLU models. While models like RoBERTa and GPT-3 achieve high accuracy, their susceptibility to simple paraphrasing suggests they often rely on surface cues rather than deep logical understanding. Future work should expand this methodology to other languages and explore the relationship between model confidence and paraphrastic robustness.

Summary updated

Abstract

We present PaRTE, a collection of 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing. We posit that if RTE models understand language, their predictions should be consistent across inputs that share the same meaning. We use the evaluation set to determine if RTE models' predictions change when examples are paraphrased. In our experiments, contemporary models change their predictions on 8-16\% of paraphrased examples, indicating that there is still room for improvement.
BibTeX
@inproceedings{verma2023evaluating,
  author = {Dhruv Verma and Yash Kumar Lal and Shreyashee Sinha and Benjamin Van Durme and Adam Poliak},
  title = {{Evaluating Paraphrastic Robustness in Textual Entailment Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=xbGP27lD1d},
}
Figure