Abstract
Traditional paraphrase identification treats sentence pairs as either identical in meaning or not. However, most "paraphrases" in standard datasets contain subtle differences that binary labels fail to capture. This paper introduces ParaTag, a dataset and framework that treats paraphrasing as a fine-grained sequence tagging task. By identifying specific non-matching spans, ParaTag enables more precise evaluation of natural language generation (NLG) and higher-quality data augmentation.
Introduction
In Natural Language Processing (NLP), paraphrase identification has long been framed as a binary classification task. Popular benchmarks like the Microsoft Research Paraphrase Corpus (MRPC) provide a single label for sentence pairs. However, researchers discovered a significant "strictness gap": 90% of pairs labeled as "paraphrases" in MRPC actually contain phrases that are not reflected in the corresponding sentence. This leads models to overlook hallucinations or factual inconsistencies because they are rewarded for being "close enough."
Methodology and Data Collection
The core innovation of ParaTag is a fine-grained annotation schema. Instead of a global label, human annotators highlight the minimum spans of tokens that cannot be paraphrased by the other sentence.
The dataset construction involved a multi-stage pipeline:
- Re-annotating MRPC: Adding word-level tags to existing paraphrase pairs to identify non-matching spans.
- GPT-3 Generation: Creating diverse candidate pairs to expand the dataset beyond existing corpora.
- Iterative Collection: Using a detection model to filter candidates before final human labeling to ensure high data quality.
The resulting ParaTag dataset consists of 30,000 sentence pairs with 840,000 word-level annotations.
Model Architectures and Applications
The authors leverage the ParaTag dataset for three primary tasks: detection, evaluation, and generation.
Paraphrase Detection
The task is framed as sequence tagging using a DeBERTa-v3 model. Every word is assigned a binary tag (0 for paraphrased, 1 for non-paraphrased). This model achieved an F1 score of 0.7063, setting a strong baseline for identifying specific semantic mismatches.
NLG Evaluation
By using the tagging model, the authors developed "PScore" (Precision) and "RScore" (Recall) metrics. These scores measure how well a generated summary matches a reference text at a granular level. This approach outperformed standard metrics like BERTScore and MoverScore by 13% on summarization tasks in terms of correlation with human judgment.
Data Augmentation
A BART-Large model was fine-tuned specifically on the "strictly paraphrased" subsets of ParaTag. Using this high-quality generator for data augmentation resulted in a 1.5% improvement on the GLUE benchmark, demonstrating that quality (strictness) is more important than quantity in paraphrase data for downstream performance.
Key Insights
- The Strictness Gap: The finding that 90% of "official" paraphrases are not strict highlights a major flaw in current benchmarks that may reward inaccurate models.
- Factuality is in the Details: ParaTag is significantly better at catching factual inconsistencies in AI-generated summaries because it focuses on specific word-level mismatches rather than overall word overlap.
- Quality Over Quantity: Training on a smaller set of verified, strict paraphrases yields better results than using massive, noisy datasets.
Conclusion and Future Work
ParaTag shifts the focus of paraphrase research from binary classification to detailed sequence tagging. While human agreement remains a challenge (Cohen’s kappa of 0.61–0.66), the dataset provides a powerful tool for improving NLG evaluation and model training. Future work may involve incorporating syntactic structures or dependency parsing to better understand the nature of non-paraphrased spans.
Summary updated