Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets

Hongren Mao, Hung-yi Lee

Abstract

This paper investigates a critical flaw in current paraphrase generation benchmarks. By introducing a "parroting" baseline—simply copying the input sentence—the authors demonstrate that standard evaluation metrics like BLEU and METEOR often reward lexical overlap over semantic creativity. Results show that this trivial baseline significantly outperforms state-of-the-art (SOTA) neural models on major datasets like Twitter and Quora, suggesting that current evaluation pipelines may be fundamentally misaligned with the goal of generating diverse paraphrases.

Introduction

Paraphrase generation is a vital task in Natural Language Processing (NLP), aimed at rewriting sentences to maintain meaning while altering structure or vocabulary. However, the field faces a "sanity check" crisis. Standard metrics such as BLEU and METEOR measure n-gram overlap between a generated output and a reference. In the context of paraphrasing, where the goal is lexical dissimilarity, these metrics may inadvertently penalize creative models while rewarding those that change nothing. This study evaluates whether "parroting"—the act of copying the input—can outperform complex deep learning architectures.

Methodology: The Parroting Baselines

The researchers proposed two unsupervised baselines to test the integrity of current benchmarks:

  1. Full Parroting: The model outputs the input sentence verbatim ($o = i$).
  2. Partial Parroting: The model introduces trivial destructive changes, such as cutting words from the start (head), end (tail), or random positions. For replacements, out-of-vocabulary words were used to intentionally lower the scores.

These baselines were compared against SOTA models, including Variational Autoencoders (VAEs) and Pointer Networks optimized via reinforcement learning, across three datasets: QUORA (question pairs), TWITTER (tweets with shared URLs), and MSCOCO (image captions).

The Single-Reference Problem

A major driver of parroting's success is the lack of reference diversity in common datasets. In the QUORA dataset, a vast majority of entries provide only a single reference paraphrase, which often remains lexically very similar to the source.

Distribution of reference paraphrases for Quora entries, showing that 64.4% of entries have only one reference.

When only one reference is available, a model that generates a valid but lexically distinct paraphrase will likely receive a lower score than a model that simply copies the input, provided the input and reference share significant n-gram overlap. This bias makes it difficult for sophisticated models to prove their worth over simple copying.

Key Results and Robustness

The performance of the parroting baseline was surprisingly dominant across the tested benchmarks:

  • Twitter: Full parroting outperformed SOTA models by +42.67% in BLEU and +106.77% in METEOR.
  • Quora: Full parroting exceeded SOTA METEOR scores by +14.88%.

Furthermore, the "partial parroting" experiments revealed the extreme resilience of these scores. Even when the input was significantly degraded, the resulting scores remained higher than those achieved by sophisticated neural models.

Metric scores on the Twitter dataset as a function of the head text cutting ratio, showing that partial parroting remains competitive even with significant data loss.

On the Twitter dataset, a model could remove nearly 40% of the input sentence and still achieve a BLEU score superior to the SOTA. Similarly, on Quora, cutting 10% of the input still resulted in a METEOR score that outperformed existing models. These findings suggest that the metrics are measuring noise and overlap rather than true paraphrastic ability.

Discussion and Conclusion

The success of parroting highlights a "metric trap" where similarity-based evaluations fail to capture the essence of paraphrasing. While MSCOCO proved less susceptible to parroting, the authors noted this was due to the captions being poor semantic matches (describing different parts of an image) rather than model superiority.

The authors urge the NLP community to move beyond n-gram overlap metrics. Future research should prioritize human evaluation or develop metrics that explicitly reward lexical diversity while maintaining semantic equivalence. Crucially, "parroting" should be adopted as a standard baseline to ensure that new models are truly learning to paraphrase rather than exploiting metric flaws.

Summary updated

Abstract

Hong-Ren Mao, Hung-Yi Lee. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
BibTeX
@inproceedings{mao2019polly,
  author = {Hongren Mao and Hung-yi Lee},
  title = {{Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=EpSLV5TeLTX},
}

Loading related content...

Figure