Towards Better Characterization of Paraphrases

Timothy Liu, De Wen Soh

Introduction

Paraphrase identification is a cornerstone of Natural Language Processing (NLP), yet the tools used to evaluate it often lack precision. Standard metrics like BLEU, ROUGE, and edit distance primarily measure surface-level overlap. They fail to distinguish between structural changes—such as moving from active to passive voice—and lexical changes, like the use of synonyms. This "bluntness" obscures the nuances of how language is transformed while preserving meaning.

Furthermore, widely used datasets like the Microsoft Research Paraphrase Corpus (MRPC) contain significant labeling noise. Models trained on such data may learn to ignore critical details like specific entities or numbers, leading to poor generalization and potential "hallucinations" in generative tasks.

Quantifying Deviation: WPD and LD

To address these limitations, researchers Timothy Liu and De Wen Soh propose two intuitive metrics designed to capture distinct dimensions of paraphrasing:

  1. Word Position Deviation (WPD): This metric quantifies structural alteration. By calculating the relative shift of common words between two sentences, WPD identifies major reorderings.
  2. Lexical Deviation (LD): This metric focuses on vocabulary differences. It measures the ratio of common words to the total unique words in a pair, highlighting when different words are used to express the same concept.
Illustration of word position shifts used to calculate Word Position Deviation (WPD).

These metrics allow for a more granular analysis of datasets. For instance, the authors found that while MRPC is lexically diverse (high LD), it is structurally simple (low WPD). Conversely, the PAWS (Paraphrase Adversaries from Word Scrambling) dataset features high structural deviation with almost no lexical change.

Cleaning the Data: MRPC-R1

The study reveals that a staggering 77% of paraphrase pairs in the original MRPC contain inconsistent entities, such as conflicting prices or names. To rectify this, the authors introduced MRPC-R1. Using Named Entity Recognition (NER), they identified these inconsistencies and either corrected the entities to match or relabeled the pairs as non-paraphrases. This "cleaned" dataset forces models to attend to the logical consistency of a paraphrase rather than just surface patterns.

Key Insights and Results

The distinction between structural and lexical shifts is critical for model performance. As shown in the examples below, WPD and LD provide a clearer picture of sentence transformations than traditional metrics like ROUGE-L, which can assign similar scores to very different types of changes.

Comparison of WPD and LD across different sentence pairs, demonstrating how WPD captures structural reordering while ROUGE-L remains relatively static.

The impact of training on high-quality, cleaned data is substantial. When a DeBERTa-large model was trained on the original MRPC and tested on the adversarial PAWS dataset, it achieved an F1 score of only 30.80. However, when trained on the revised MRPC-R1, the model's performance on PAWS jumped to 55.22 F1. By fixing labels and entities, the researchers nearly doubled the model's ability to generalize to complex structural variations.

Conclusion

WPD and LD offer a path toward more controllable and interpretable paraphrase identification and generation. While these metrics are currently "coarse"—they do not yet distinguish between specific linguistic transformations like active-to-passive voice—they provide a necessary foundation for better dataset characterization. Future work may integrate these metrics into loss functions to drive the development of more robust and reliable NLP models.

Summary updated

Abstract

To effectively characterize the nature of paraphrase pairs without expert human annotation, we proposes two new metrics: word position deviation (WPD) and lexical deviation (LD). WPD measures the degree of structural alteration, while LD measures the difference in vocabulary used. We apply these metrics to better understand the commonly-used MRPC dataset and study how it differs from PAWS, another paraphrase identification dataset. We also perform a detailed study on MRPC and propose improvements to the dataset, showing that it improves generalizability of models trained on the dataset. Lastly, we apply our metrics to filter the output of a paraphrase generation model and show how it can be used to generate specific forms of paraphrases for data augmentation or robustness testing of NLP models.
BibTeX
@inproceedings{liu2022towards,
  author = {Timothy Liu and De Wen Soh},
  title = {{Towards Better Characterization of Paraphrases}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.588},
}
Figure