Perplexity from PLM Is Unreliable for Evaluating Text Quality

Yequan Wang, Jiawen Deng, Aixin Sun, Xuying MengarXiv 2022 · DOI · arXiv

Introduction

In the evaluation of natural language generation (NLG), researchers frequently employ Perplexity (PPL) as a proxy for text fluency. Calculated using pre-trained language models (PLMs) like GPT-2, the underlying assumption is that high-quality, fluent text will be assigned a high probability (and thus low perplexity) by a well-trained model. However, recent research suggests this metric is fundamentally flawed and easily manipulated by structural quirks of the models themselves.

The Problem with Perplexity

While metrics like BLEU and ROUGE focus on reference overlap, PPL was intended to capture the inherent "naturalness" of generated text. The study "Perplexity from PLM Is Unreliable for Evaluating Text Quality" investigates whether low PPL truly correlates with human-perceived quality. The findings reveal that PPL is biased toward specific structural patterns—such as length and repetition—that do not align with human standards of good writing.

Methodology

The researchers used GPT2-large to evaluate sentences from the WikiText-2 test split, which consists of high-quality, human-written Wikipedia articles. They systematically modified these "gold standard" sentences to test three hypotheses:

  1. Length Sensitivity: Does sentence length inherently affect PPL?
  2. Repetition Bias: Does the common generative error of "looping" lower PPL?
  3. Punctuation Sensitivity: How do minor formatting changes impact the score?

Key Findings

1. The Length Bias

PPL is not length-invariant. Analysis showed that shorter sentences (under 25 tokens) exhibit significantly higher variance and generally higher PPL scores than longer sequences. This mathematical artifact suggests that models may unfairly penalize concise, high-quality text in favor of longer, potentially redundant sequences.

2. The Repetition Paradox

The most significant finding is that PPL rewards mindless repetition. In human evaluation, repeating a phrase makes text worse. However, because autoregressive models become increasingly "certain" of a token the more it appears, PPL drops as repetition increases.

Figure 2: PPL scores decrease as text is repeated (2-times, 3-times, 4-times), showing a bias toward redundant sequences.

As shown in Figure 2, increasing the number of repetitions consistently lowers the PPL across various lengths, effectively rewarding the exact "looping" behavior that researchers aim to eliminate in generative models.

3. Fragility to Punctuation

PPL is hyper-sensitive to minor formatting. Removing a single trailing period can cause a massive spike in PPL. For example, removing the last punctuation mark increased the average PPL from 169.3 to 279.5, despite having negligible impact on human readability.

Key Results and Examples

The quantitative impact of these biases is stark. In controlled experiments, repeating the last 10 tokens of a sentence 12 times dropped the average PPL from 411.99 to 4.84.

Figure 6: Examples of text modifications and their resulting PPL scores. Red values indicate significant increases in PPL due to minor changes like punctuation removal.

Figure 6 illustrates how specific modifications lead to PPL scores that contradict human intuition. In ID 5, removing punctuation caused the PPL to skyrocket to over 50,000, while repetition (2-Times) significantly lowered the score compared to the original high-quality sentence.

Conclusion

The study concludes that PPL is an unreliable standalone metric for text quality. It is fundamentally biased toward longer sequences, rewards redundant repetition, and is overly sensitive to non-semantic formatting like punctuation. Future evaluation frameworks should move toward length-normalized metrics that penalize repetition and prioritize text diversity over local token probability.

Summary updated

Abstract

Recently, amounts of works utilize perplexity~(PPL) to evaluate the quality of the generated text. They suppose that if the value of PPL is smaller, the quality(i.e. fluency) of the text to be evaluated is better. However, we find that the PPL referee is unqualified and it cannot evaluate the generated text fairly for the following reasons: (i) The PPL of short text is larger than long text, which goes against common sense, (ii) The repeated text span could damage the performance of PPL, and (iii) The punctuation marks could affect the performance of PPL heavily. Experiments show that the PPL is unreliable for evaluating the quality of given text. Last, we discuss the key problems with evaluating text quality using language models.
BibTeX
@misc{wang2022perplexity,
  author = {Yequan Wang and Jiawen Deng and Aixin Sun and Xuying Meng},
  title = {{Perplexity from PLM Is Unreliable for Evaluating Text Quality}},
  year = {2022},
  eprint = {2210.05892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.05892},
}
Figure