Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets

Hongren Mao, Hung-yi Lee

Abstract

Paraphrase generation is an interesting and challenging NLP task which has numerous practical applications. In this paper, we analyze datasets commonly used for paraphrase generation research, and show that simply parroting input sentences surpasses state-of-the-art models in the literature when evaluated on standard metrics. Our findings illustrate that a model could be seemingly adept at generating paraphrases, despite only making trivial changes to the input sentence or even none at all.

1. Introduction

Paraphrase generation—the task of restating a sentence while preserving its semantic meaning—is a cornerstone of natural language processing (NLP). It finds utility in data augmentation, question-answering systems, and dialogue generation. Traditionally, researchers evaluate these models using metrics adapted from machine translation, specifically BLEU, METEOR, and TER. These metrics primarily quantify n-gram overlap between a generated output and human-written reference paraphrases.

However, this evaluation paradigm contains a critical "blind spot." If the reference paraphrases in a dataset are lexically similar to the original input, a model that simply copies the input (an "identity function") will receive a high score. This paper investigates whether current state-of-the-art (SOTA) models are performing meaningful linguistic transformations or if they are being outpaced by a simple "parroting" baseline.

2. Methodology

To challenge complex neural architectures, we propose two unsupervised baselines:

  1. Full Parroting: The model outputs the input sentence exactly ($o = i$).
  2. Partial Parroting: The model modifies the input by removing or replacing words from the beginning (head), end (tail), or random positions.

These baselines require no training or specialized hardware. We compare their performance against published SOTA results on three benchmarks: QUORA (question pairs), TWITTER (tweets sharing a URL), and MSCOCO (image captions).

3. Experimental Results

3.1 The Lethal Identity Baseline

In many instances, the simple identity function outperformed complex neural models. On the TWITTER dataset, full parroting surpassed the SOTA by 106.77% on the METEOR metric and 42.67% on BLEU. Similarly, on QUORA, full parroting beat the SOTA METEOR score (38.60 vs. 33.6, a 14.88% improvement) and remained highly competitive on BLEU.

3.2 Robustness of Partial Parroting

Even when the input was intentionally degraded, the resulting scores remained remarkably high. As shown in the figures below, cutting a significant portion of the input text (the "head text cutting ratio") still yielded scores that rival or exceed supervised models.

Metric scores on the Twitter dataset as a function of the head text cutting ratio. Full parroting (ratio 0) significantly outperforms SOTA benchmarks.
Metric scores on the Quora dataset as a function of the head text cutting ratio. Even with 20% of the text removed, scores remain high.

3.3 Dataset Variance

MSCOCO proved to be the exception. Parroting performed poorly here because captions for the same image are often semantically related but structurally diverse (e.g., "A dog runs" vs. "A brown canine plays"). This suggests MSCOCO requires actual semantic understanding, though its diversity may also introduce noise for pure paraphrase tasks.

4. Discussion and Implications

Our results serve as a wake-up call for the NLP community. The high performance of parroting suggests that current metrics reward lexical laziness rather than linguistic creativity. If a model uses a valid synonym not present in the reference, its score may drop, whereas copying the input is heavily rewarded due to high n-gram overlap.

Furthermore, dataset bias exacerbates this issue. In the QUORA dataset, over 64% of entries have only a single reference paraphrase. If that reference is similar to the input, the "parroting" effect is amplified, making it a viable strategy for climbing leaderboards without achieving true paraphrastic capability.

5. Conclusion

We do not claim that parroting is a superior method for paraphrasing in a human sense; an identity function is not a paraphrase. Rather, we highlight that our current evaluation tools and datasets are flawed. Future research must prioritize:

  • Qualitative Analysis: Moving beyond automated metrics to verify actual text transformation.
  • Diverse Metrics: Developing metrics that reward semantic similarity while penalizing lexical copying.
  • High-Quality Datasets: Creating benchmarks with more diverse, high-quality reference paraphrases to ensure that "parroting" is no longer a successful strategy.

Summary updated

Abstract

Paraphrase generation is an interesting and challenging NLP task which has numerous practical applications. In this paper, we analyze datasets commonly used for paraphrase generation research, and show that simply parroting input sentences surpasses state-of-the-art models in the literature when evaluated on standard metrics. Our findings illustrate that a model could be seemingly adept at generating paraphrases, despite only making trivial changes to the input sentence or even none at all.
BibTeX
@inproceedings{mao2019polly,
  author = {Hongren Mao and Hung-yi Lee},
  title = {{Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=EpSLV5TeLTX},
}

Loading related content...

Figure