Abstract
Paraphrase generation is an interesting and challenging NLP task which has numerous practical applications. In this paper, we analyze datasets commonly used for paraphrase generation research, and show that simply parroting input sentences surpasses state-of-the-art models in the literature when evaluated on standard metrics. Our findings illustrate that a model could be seemingly adept at generating paraphrases, despite only making trivial changes to the input sentence or even none at all.
1. Introduction
Paraphrase generation—the task of restating a sentence while preserving its semantic meaning—is a cornerstone of natural language processing (NLP). It finds utility in data augmentation, question-answering systems, and dialogue generation. Traditionally, researchers evaluate these models using metrics adapted from machine translation, specifically BLEU, METEOR, and TER. These metrics primarily quantify n-gram overlap between a generated output and human-written reference paraphrases.
However, this evaluation paradigm contains a critical "blind spot." If the reference paraphrases in a dataset are lexically similar to the original input, a model that simply copies the input (an "identity function") will receive a high score. This paper investigates whether current state-of-the-art (SOTA) models are performing meaningful linguistic transformations or if they are being outpaced by a simple "parroting" baseline.
2. Methodology
To challenge complex neural architectures, we propose two unsupervised baselines:
- Full Parroting: The model outputs the input sentence exactly ($o = i$).
- Partial Parroting: The model modifies the input by removing or replacing words from the beginning (head), end (tail), or random positions.
These baselines require no training or specialized hardware. We compare their performance against published SOTA results on three benchmarks: QUORA (question pairs), TWITTER (tweets sharing a URL), and MSCOCO (image captions).
3. Experimental Results
3.1 The Lethal Identity Baseline
In many instances, the simple identity function outperformed complex neural models. On the TWITTER dataset, full parroting surpassed the SOTA by 106.77% on the METEOR metric and 42.67% on BLEU. Similarly, on QUORA, full parroting beat the SOTA METEOR score (38.60 vs. 33.6, a 14.88% improvement) and remained highly competitive on BLEU.
3.2 Robustness of Partial Parroting
Even when the input was intentionally degraded, the resulting scores remained remarkably high. As shown in the figures below, cutting a significant portion of the input text (the "head text cutting ratio") still yielded scores that rival or exceed supervised models.
3.3 Dataset Variance
MSCOCO proved to be the exception. Parroting performed poorly here because captions for the same image are often semantically related but structurally diverse (e.g., "A dog runs" vs. "A brown canine plays"). This suggests MSCOCO requires actual semantic understanding, though its diversity may also introduce noise for pure paraphrase tasks.
4. Discussion and Implications
Our results serve as a wake-up call for the NLP community. The high performance of parroting suggests that current metrics reward lexical laziness rather than linguistic creativity. If a model uses a valid synonym not present in the reference, its score may drop, whereas copying the input is heavily rewarded due to high n-gram overlap.
Furthermore, dataset bias exacerbates this issue. In the QUORA dataset, over 64% of entries have only a single reference paraphrase. If that reference is similar to the input, the "parroting" effect is amplified, making it a viable strategy for climbing leaderboards without achieving true paraphrastic capability.
5. Conclusion
We do not claim that parroting is a superior method for paraphrasing in a human sense; an identity function is not a paraphrase. Rather, we highlight that our current evaluation tools and datasets are flawed. Future research must prioritize:
- Qualitative Analysis: Moving beyond automated metrics to verify actual text transformation.
- Diverse Metrics: Developing metrics that reward semantic similarity while penalizing lexical copying.
- High-Quality Datasets: Creating benchmarks with more diverse, high-quality reference paraphrases to ensure that "parroting" is no longer a successful strategy.
Summary updated