Abstract
Automatic evaluation for open-ended natural language generation tasks remains a challenge. Existing metrics such as BLEU show a low correlation with human judgment. We propose a novel and powerful learning-based evaluation metric: Perception Score. The method measures the overall quality of the generation and scores holistically instead of only focusing on one evaluation criteria, such as word overlapping. Moreover, it also shows the amount of uncertainty about its evaluation result. By connecting the uncertainty, Perception Score gives a more accurate evaluation for the generation system. Perception Score provides state-of-the-art results on two conditional generation tasks and two unconditional generation tasks.
Introduction
Evaluating creative, open-ended text generation (like storytelling) is notoriously difficult. Traditional automated metrics like BLEU and ROUGE were designed for machine translation, where the goal is to stay close to a specific reference. However, in open-ended generation—such as finishing a story or writing a movie review—there are thousands of "correct" ways to respond.
A model might generate a brilliant, creative sentence that shares zero words with the human reference; traditional metrics would give this a score of zero. Even newer neural metrics like BERTScore often fail because they do not account for the context of the prompt or the inherent uncertainty of creative writing. The authors aim to bridge this gap by creating a metric that evaluates "holistic quality" rather than just surface-level similarity.
Methodology
The authors propose Perception Score, which treats evaluation as a distribution-matching problem. Instead of comparing sentences point-by-point, the metric learns to measure the distance between the distribution of generated text ($P_G$) and the distribution of real human references ($P_R$).
Architecture and Contextual Awareness
Unlike many predecessors, Perception Score incorporates the prompt/context. It uses a pre-trained RoBERTa model to encode the triplet of [Context; SEP; Generation] and [Context; SEP; Reference]. This allows the metric to judge if a sentence is appropriate for the situation, not just if it is grammatically correct.
Uncertainty Estimation
A key innovation of this approach is the calculation of two types of uncertainty:
- Model Uncertainty: Measured via Monte Carlo Dropout (running the model multiple times with different neurons deactivated to see how much the score fluctuates).
- Data Uncertainty: A learned parameter that identifies samples that are inherently ambiguous or noisy.
The final system-level score is not a simple average; it weights individual samples based on the model's confidence, ensuring that the final evaluation is driven by the generations the metric is most "sure" about.
Key Results
The authors validated the metric on four distinct tasks: ROCStories (story completion), IMDB (conditional and unconditional movie reviews), and COCO Image Captions.
Superior Correlation
Perception Score achieved the highest Pearson correlation with human judgment (Turing Score M1) across all four tasks. On the ROCStories dataset, it reached a correlation of 0.692, dwarfing BLEU-4 (0.355) and even the popular BERTScore (0.282). Even the "base" version of Perception Score outperformed the "large" versions of BERTScore and MoverScore, suggesting the architectural approach is more important than raw parameter count.
Adversarial Robustness
In tests where humans intentionally wrote "adversarial" endings (sentences that used the right words but were logically wrong), traditional metrics like BLEU and BERTScore gave them high scores. Perception Score was the only metric that correctly identified these as low-quality, demonstrating its ability to capture "realness" over simple word overlap.
Limitations and Future Work
While powerful, Perception Score has a few drawbacks:
- Computational Overhead: Relying on a large transformer and multiple passes for uncertainty estimation makes it slower and more expensive than n-gram metrics.
- Domain Specificity: The metric needs to be fine-tuned on the specific task it is evaluating; it is not a "plug-and-play" metric like BLEU.
Future directions include exploring how this metric could be used as a reward function in Reinforcement Learning to directly train models to generate more "perceptually" pleasing text.
Summary updated