Introduction
Paraphrase identification is a cornerstone of Natural Language Processing (NLP), yet the tools used to evaluate it often lack precision. Standard metrics like BLEU, ROUGE, and edit distance primarily measure surface-level overlap. They fail to distinguish between structural changes—such as moving from active to passive voice—and lexical changes, like the use of synonyms. This "bluntness" obscures the nuances of how language is transformed while preserving meaning.
Furthermore, widely used datasets like the Microsoft Research Paraphrase Corpus (MRPC) contain significant labeling noise. Models trained on such data may learn to ignore critical details like specific entities or numbers, leading to poor generalization and potential "hallucinations" in generative tasks.
Quantifying Deviation: WPD and LD
To address these limitations, researchers Timothy Liu and De Wen Soh propose two intuitive metrics designed to capture distinct dimensions of paraphrasing:
- Word Position Deviation (WPD): This metric quantifies structural alteration. By calculating the relative shift of common words between two sentences, WPD identifies major reorderings.
- Lexical Deviation (LD): This metric focuses on vocabulary differences. It measures the ratio of common words to the total unique words in a pair, highlighting when different words are used to express the same concept.
These metrics allow for a more granular analysis of datasets. For instance, the authors found that while MRPC is lexically diverse (high LD), it is structurally simple (low WPD). Conversely, the PAWS (Paraphrase Adversaries from Word Scrambling) dataset features high structural deviation with almost no lexical change.
Cleaning the Data: MRPC-R1
The study reveals that a staggering 77% of paraphrase pairs in the original MRPC contain inconsistent entities, such as conflicting prices or names. To rectify this, the authors introduced MRPC-R1. Using Named Entity Recognition (NER), they identified these inconsistencies and either corrected the entities to match or relabeled the pairs as non-paraphrases. This "cleaned" dataset forces models to attend to the logical consistency of a paraphrase rather than just surface patterns.
Key Insights and Results
The distinction between structural and lexical shifts is critical for model performance. As shown in the examples below, WPD and LD provide a clearer picture of sentence transformations than traditional metrics like ROUGE-L, which can assign similar scores to very different types of changes.
The impact of training on high-quality, cleaned data is substantial. When a DeBERTa-large model was trained on the original MRPC and tested on the adversarial PAWS dataset, it achieved an F1 score of only 30.80. However, when trained on the revised MRPC-R1, the model's performance on PAWS jumped to 55.22 F1. By fixing labels and entities, the researchers nearly doubled the model's ability to generalize to complex structural variations.
Conclusion
WPD and LD offer a path toward more controllable and interpretable paraphrase identification and generation. While these metrics are currently "coarse"—they do not yet distinguish between specific linguistic transformations like active-to-passive voice—they provide a necessary foundation for better dataset characterization. Future work may integrate these metrics into loss functions to drive the development of more robust and reliable NLP models.
Summary updated