Paraphrase Identification via Textual Inference

Ning Shi, Bradley Hauer, Jai Riley, Grzegorz KondrakNAACL 2024

TL;DR

This paper proposes a novel approach to paraphrase identification (PI) by reframing it as a natural language inference (NLI) problem. By formalizing the relationship between these tasks and developing a dataset adaptation method, they demonstrate that NLI models, when fine-tuned, can outperform dedicated PI models on benchmark PI datasets, offering a more robust and theoretically grounded way to tackle paraphrase detection.

What Problem Does This Paper Address?

Paraphrase identification (PI) and natural language inference (NLI) are two fundamental tasks in natural language processing, both concerned with understanding semantic relationships between sentences. PI aims to determine if two sentences convey the same meaning, while NLI classifies the relationship between a premise and a hypothesis into entailment, contradiction, or neutral. Despite their distinct objectives, the underlying connection between these tasks has been underexplored. Existing PI methods often rely on specific architectures and may struggle with nuanced semantic variations. This paper seeks to bridge this gap by proposing a unified theoretical framework and a practical method to leverage the power of NLI models for PI, aiming for improved performance and a deeper understanding of semantic relations.

Methods

The core of this paper's contribution lies in its theoretical formalization and a novel dataset adaptation procedure.

Theoretical Framework: The authors define four semantic relations: semantic equivalence (SEQ), paraphrase relation (PR), textual entailment (TE), and textual inference (TI). They formalize the relationship between these, notably postulating that semantic equivalence is bidirectional textual entailment (SEQ(S1, S2) ⇔ TE(S1, S2) ∧ TE(S2, S1)), and that paraphrasing is a contextual generalization of this bidirectional entailment (PR(C, S1, S2) ⇔ TI(C, S1, S2) ∧ TI(C, S2, S1)). This theoretical grounding implies a reduction of PI to NLI.

Dataset Adaptation for PI2NLI: To implement this reduction, they propose a method to convert PI datasets into NLI instances. Positive PI instances are transformed into two directional NLI instances (mutual TI). For negative PI instances, a single NLI instance is created in a random direction, labeled as "neutral" or "contradiction." This process aims to adapt PI data for fine-tuning NLI models, mitigating potential biases from transfer learning and domain-specific data characteristics.

Zero-shot and Fine-tuned Evaluation: They evaluate their approach, termed PI2NLI, in two settings:

  • Zero-shot (pi2nlizero): Pre-trained NLI models are directly applied to PI tasks without fine-tuning on adapted data.
  • Fine-tuned (pi2nli): NLI models are fine-tuned on the PI datasets converted to NLI format using their adaptation procedure.

Datasets

The paper evaluates their method on six standard PI benchmarks:

  • PIT: Paraphrase and Semantic Similarity in Twitter
  • QQP: Quora Question Pairs
  • MSRP: Microsoft Research Paraphrase Corpus
  • PAWS QQP: Paraphrase Adversaries from Word Scrambling (QQP subset)
  • PAWS Wiki: Paraphrase Adversaries from Word Scrambling (Wikipedia subset)
  • PARADE: PARAphrase identification based on Domain knowledgE

The paper also mentions using NLI datasets for fine-tuning, though specific NLI datasets used for pre-training are not explicitly detailed beyond referring to models trained on "SNLI, MNLI, FEVER, ANLI."

Key Insights and Ideas

  • PI as Bidirectional TI: The formalization of paraphrase identification as bidirectional textual inference provides a strong theoretical basis for using NLI models, which are designed for unidirectional inference, to solve PI.
  • Dataset Adaptation is Crucial: The zero-shot performance of PI2NLI is erratic, highlighting inherent flaws and inconsistencies in existing PI benchmarks. Fine-tuning with their dataset adaptation method significantly improves performance, suggesting that models need to learn the specific context and nuances of each PI dataset.
  • NLI Models Can Outperform Dedicated PI Models: The fine-tuned PI2NLI approach, using standard NLI models, achieves state-of-the-art results on several PI benchmarks, even surpassing dedicated PI models that use the same underlying language models. This underscores the power of the PI2NLI reduction and the effectiveness of their fine-tuning strategy.
  • Pre-training on NLI Data Benefits PI: The observation that pre-training on additional NLI data leads to better and more stable fine-tuned performance on PI tasks suggests a synergistic relationship between NLI and PI, where skills learned in NLI transfer effectively to PI.

Key Results

  • Fine-tuned PI2NLI achieves state-of-the-art: The fine-tuned version of their PI2NLI reduction consistently yields high F1 scores across all six PI benchmarks, outperforming prior work. For instance, the F1 score for the ROBERTalarge-based PI2NLI on the PIT dataset increased from 10.70 to 83.64.
  • Outperforms dedicated PI models: On four out of six datasets, PI2NLI consistently outperforms dedicated PI models using the same language models.
  • Zero-shot performance is variable: The zero-shot PI2NLI approach shows highly variable F1 scores, outperforming random baselines on only half of the datasets, indicating issues with the PI benchmarks themselves.
  • Improved accuracy balance: Figure 2 shows that fine-tuning (b) pi2nli leads to a more balanced and increased positive and negative accuracy compared to the zero-shot setting (a) pi2nlizero, suggesting models adjust their decision boundaries effectively.

Limitations and Future Work

Limitations: * Zero-shot benchmark issues: The erratic zero-shot performance highlights that current PI benchmarks may not adequately capture the bidirectional inference relation required for genuine paraphrase identification. * NLI-centric approach: The study is primarily NLI-centric, and further exploration from a PI-centric perspective (e.g., using PI models for NLI tasks) could offer a more balanced understanding. * Dataset adaptation for negative instances: The heuristic of randomly assigning "neutral" or "contradiction" for negative NLI instances is not theoretically justified and could be improved with more sophisticated methods, potentially using pre-trained NLI models for finer-grained annotation.

Future Work: * Augmenting PI datasets with NLI data: Exploring how augmenting PI datasets with NLI data could yield new insights. * PI-centric exploration: Investigating the capability of PI models in solving NLI tasks to achieve a more comprehensive understanding of semantic relations. * Refining negative instance handling: Developing more advanced methods for converting negative PI instances to NLI format.

Summary updated

BibTeX
@inproceedings{shi2024paraphrase,
  author = {Ning Shi and Bradley Hauer and Jai Riley and Grzegorz Kondrak},
  title = {{Paraphrase Identification via Textual Inference}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure