Introduction
In the evaluation of natural language generation (NLG), researchers frequently employ Perplexity (PPL) as a proxy for text fluency. Calculated using pre-trained language models (PLMs) like GPT-2, the underlying assumption is that high-quality, fluent text will be assigned a high probability (and thus low perplexity) by a well-trained model. However, recent research suggests this metric is fundamentally flawed and easily manipulated by structural quirks of the models themselves.
The Problem with Perplexity
While metrics like BLEU and ROUGE focus on reference overlap, PPL was intended to capture the inherent "naturalness" of generated text. The study "Perplexity from PLM Is Unreliable for Evaluating Text Quality" investigates whether low PPL truly correlates with human-perceived quality. The findings reveal that PPL is biased toward specific structural patterns—such as length and repetition—that do not align with human standards of good writing.
Methodology
The researchers used GPT2-large to evaluate sentences from the WikiText-2 test split, which consists of high-quality, human-written Wikipedia articles. They systematically modified these "gold standard" sentences to test three hypotheses:
- Length Sensitivity: Does sentence length inherently affect PPL?
- Repetition Bias: Does the common generative error of "looping" lower PPL?
- Punctuation Sensitivity: How do minor formatting changes impact the score?
Key Findings
1. The Length Bias
PPL is not length-invariant. Analysis showed that shorter sentences (under 25 tokens) exhibit significantly higher variance and generally higher PPL scores than longer sequences. This mathematical artifact suggests that models may unfairly penalize concise, high-quality text in favor of longer, potentially redundant sequences.
2. The Repetition Paradox
The most significant finding is that PPL rewards mindless repetition. In human evaluation, repeating a phrase makes text worse. However, because autoregressive models become increasingly "certain" of a token the more it appears, PPL drops as repetition increases.
As shown in Figure 2, increasing the number of repetitions consistently lowers the PPL across various lengths, effectively rewarding the exact "looping" behavior that researchers aim to eliminate in generative models.
3. Fragility to Punctuation
PPL is hyper-sensitive to minor formatting. Removing a single trailing period can cause a massive spike in PPL. For example, removing the last punctuation mark increased the average PPL from 169.3 to 279.5, despite having negligible impact on human readability.
Key Results and Examples
The quantitative impact of these biases is stark. In controlled experiments, repeating the last 10 tokens of a sentence 12 times dropped the average PPL from 411.99 to 4.84.
Figure 6 illustrates how specific modifications lead to PPL scores that contradict human intuition. In ID 5, removing punctuation caused the PPL to skyrocket to over 50,000, while repetition (2-Times) significantly lowered the score compared to the original high-quality sentence.
Conclusion
The study concludes that PPL is an unreliable standalone metric for text quality. It is fundamentally biased toward longer sequences, rewards redundant repetition, and is overly sensitive to non-semantic formatting like punctuation. Future evaluation frameworks should move toward length-normalized metrics that penalize repetition and prioritize text diversity over local token probability.
Summary updated