Introduction
Human evaluation remains the "gold standard" for assessing Natural Language Generation (NLG) systems. However, obtaining consistent and reliable human judgments is notoriously difficult. Traditional methods, such as Likert scales, often suffer from low inter-annotator agreement and a "halo effect" where distinct criteria like naturalness and informativeness are conflated. We present RankME (Rank-based Magnitude Estimation), a novel evaluation framework designed to address these inconsistencies by combining continuous scales with relative assessments.
The Challenge of Human Evaluation
The subjectivity of human judgment leads to three primary issues in NLG evaluation:
- Inconsistency: Different raters often provide wildly different scores for the same output, and individual raters may be inconsistent over time.
- Criterion Conflation: Judges often struggle to separate different aspects of quality. For instance, a fluent sentence (high naturalness) might be rated as highly accurate (high informativeness) even if it contains factual errors.
- Absolute vs. Relative Assessment: Humans are naturally better at comparing items than assigning absolute numerical values in isolation. Discrete scales like 1-5 force judges to "bucket" their perceptions, losing granular information.
The RankME Framework
RankME integrates three core components to improve rating quality:
- Magnitude Estimation (ME): Judges compare target sentences to a fixed "reference" sentence, providing scores relative to that reference.
- Relative Ranking: Judges rank all target sentences simultaneously, forcing them to consider systems in relation to one another.
- Continuous Scales: Instead of discrete points, RankME uses a continuous slider without numerical labels, allowing for more nuanced and discriminative judgments.
The framework was tested using outputs from the E2E NLG Challenge, focusing on restaurant descriptions. Three systems were evaluated: Sheffield NLP (seq2seq with attention), TGen (baseline with semantic reranker), and Slug2Slug (ensemble model). The researchers compared RankME against traditional Likert scales and Plain Magnitude Estimation across two experimental setups. In Setup 1, judges evaluated all criteria simultaneously. In Setup 2, each criterion was evaluated in an isolated task to minimize cross-criteria interference.
Results and Discussion
Improved Reliability
A key metric for evaluation quality is the Intra-class Correlation Coefficient (ICC), which measures the level of agreement among raters. RankME significantly outperformed traditional methods, particularly when criteria were evaluated separately in Setup 2.
As shown in the results, RankME achieved an ICC of 0.42 for Naturalness in Setup 2, a substantial improvement over the 0.27 achieved by plain Magnitude Estimation. This suggests that the combination of relative ranking and continuous scales helps raters maintain a more consistent internal standard.
Discriminative Power and System Rankings
Beyond reliability, an evaluation method must be able to distinguish between systems of similar quality. RankME proved to be more discriminative than Likert scales, which often resulted in statistical "ties."
For the "Quality" criterion, Likert scales failed to distinguish between the Slug2Slug and TGen systems. In contrast, RankME provided a clear, statistically significant ranking of all three systems. This increased sensitivity is crucial for fine-grained error analysis and model development.
Decoupling Criteria
The study also confirmed the existence of the "halo effect." When evaluated together (Setup 1), naturalness and informativeness scores were heavily correlated. By separating these tasks (Setup 2), the researchers successfully decoupled the criteria. This allows developers to identify specific weaknesses—for example, a system that produces very natural-sounding text but fails to include all required information.
Conclusion
RankME offers a more reliable and discriminative alternative to traditional NLG evaluation methods. By leveraging human strengths in relative comparison and providing continuous scales, it produces more consistent data for benchmarking AI systems. Furthermore, when combined with Bayesian estimation techniques like TrueSkill, RankME provides a cost-effective path toward stable and trustworthy system rankings.
Summary updated