Abstract
Automatic evaluation for open-ended natural language generation (NLG) tasks remains a challenge. We propose a learned evaluation metric: Perception Score. It utilizes a pre-trained model and considers context information for conditional generation. Perception Score assigns a holistic score along with an uncertainty measurement. We conduct experiments on three open-ended conditional generation tasks and two open-ended unconditional generation tasks. Perception Score achieves state-of-the-art results on all tasks consistently in terms of correlation with human evaluation scores.
Introduction: The Challenge of Open-ended Evaluation
Evaluating open-ended natural language generation (NLG) is notoriously difficult. Traditional metrics like BLEU or ROUGE rely on n-gram overlap—counting how many words the machine's output shares with a human reference. However, in open-ended tasks like "finish this story" or "reply to this chat," there are infinite "correct" answers that might not share a single word with the reference.
While newer embedding-based metrics (like BERTScore) look at semantic similarity, they often ignore the context (the conversation history or the start of the story). Without context, a metric cannot determine if a response is actually relevant. Furthermore, most metrics provide a "black box" score without indicating how confident they are in that assessment.
Methodology: Architecture and Uncertainty
Perception Score treats evaluation as a learning problem rather than a counting problem. Its architecture consists of three main pillars:
- Contextual Representation: Unlike metrics that only look at the generated sentence vs. the reference, Perception Score concatenates the context (e.g., previous dialogue turns) with the sentences. This allows the model to "understand" if the generation fits the situation.
- Earth Mover’s Distance (EMD) Approximation: The metric is trained to distinguish between a generated sentence and a reference sentence by approximating the EMD (Wasserstein distance). It essentially learns a function that perceives the "realness" of a generation relative to human-written text.
- Uncertainty-Aware Weighting: The system calculates two types of uncertainty for every sample: Data Uncertainty (how ambiguous the specific text sample is) and Model Uncertainty (measured via Monte Carlo dropout). The final system-level score is a weighted average where samples with lower uncertainty are given more "vote" in the final quality score.
Experimental Evaluation
The researchers validated Perception Score across five diverse tasks:
- Conditional Generation: DailyDialog (chit-chat), ROCStories (story ending generation), and Large Movie Review Conditional (completing a movie review).
- Unconditional Generation: COCO Image Captions and Large Movie Review Unconditional.
These datasets represent a mix of short-form dialogue and longer-form creative writing, providing a rigorous test for any metric.
Results and Key Insights
Perception Score consistently achieved the highest correlation with human scores (Turing score M1) across all tasks.
- Superior Correlation: In the ROCStories task, Perception Score (Large) achieved a correlation of 0.692, significantly higher than BLEURT (0.529) and far beyond BLEU-4 (0.355).
- Robustness to Adversaries: The metric was tested against "Adversarial Endings" (sentences that look like the reference but make no sense in context). While BLEU and BERTScore were easily fooled, Perception Score correctly assigned them low scores.
- Efficiency vs. Scale: Interestingly, the "Base" version of Perception Score (12 layers) outperformed the "Large" versions (24 layers) of competing metrics like BERTScore and MoverScore. This suggests that the architecture (context + uncertainty) is more important than raw parameter count.
Conclusion and Future Work
Perception Score demonstrates that context is king in NLG evaluation. By feeding context into a RoBERTa backbone, the metric can penalize "hallucinations" or irrelevant responses that n-gram metrics would miss. The addition of uncertainty estimation makes the metric more robust, ensuring that ambiguous samples do not disproportionately affect the overall system evaluation.
Future work will address limitations in long-form text (50–200 words), where common-sense nuances remain challenging for pre-trained models. Additionally, the authors suggest exploring "zero-shot" versions that can evaluate new types of text generation without needing a dedicated training phase.
Summary updated