RankME: Reliable Human Ratings for Natural Language Generation

Jekaterina Novikova, Ondrej Dusek, Verena Rieser

Introduction

Human evaluation remains the "gold standard" for assessing Natural Language Generation (NLG) systems. However, obtaining consistent and reliable human judgments is notoriously difficult. Traditional methods, such as Likert scales, often suffer from low inter-annotator agreement and a "halo effect" where distinct criteria like naturalness and informativeness are conflated. We present RankME (Rank-based Magnitude Estimation), a novel evaluation framework designed to address these inconsistencies by combining continuous scales with relative assessments.

The Challenge of Human Evaluation

The subjectivity of human judgment leads to three primary issues in NLG evaluation:

  1. Inconsistency: Different raters often provide wildly different scores for the same output, and individual raters may be inconsistent over time.
  2. Criterion Conflation: Judges often struggle to separate different aspects of quality. For instance, a fluent sentence (high naturalness) might be rated as highly accurate (high informativeness) even if it contains factual errors.
  3. Absolute vs. Relative Assessment: Humans are naturally better at comparing items than assigning absolute numerical values in isolation. Discrete scales like 1-5 force judges to "bucket" their perceptions, losing granular information.

The RankME Framework

RankME integrates three core components to improve rating quality:

  • Magnitude Estimation (ME): Judges compare target sentences to a fixed "reference" sentence, providing scores relative to that reference.
  • Relative Ranking: Judges rank all target sentences simultaneously, forcing them to consider systems in relation to one another.
  • Continuous Scales: Instead of discrete points, RankME uses a continuous slider without numerical labels, allowing for more nuanced and discriminative judgments.

The framework was tested using outputs from the E2E NLG Challenge, focusing on restaurant descriptions. Three systems were evaluated: Sheffield NLP (seq2seq with attention), TGen (baseline with semantic reranker), and Slug2Slug (ensemble model). The researchers compared RankME against traditional Likert scales and Plain Magnitude Estimation across two experimental setups. In Setup 1, judges evaluated all criteria simultaneously. In Setup 2, each criterion was evaluated in an isolated task to minimize cross-criteria interference.

Results and Discussion

Improved Reliability

A key metric for evaluation quality is the Intra-class Correlation Coefficient (ICC), which measures the level of agreement among raters. RankME significantly outperformed traditional methods, particularly when criteria were evaluated separately in Setup 2.

Intra-class Correlation Coefficient (ICC) values across different methods and setups. RankME shows superior reliability in Setup 2.

As shown in the results, RankME achieved an ICC of 0.42 for Naturalness in Setup 2, a substantial improvement over the 0.27 achieved by plain Magnitude Estimation. This suggests that the combination of relative ranking and continuous scales helps raters maintain a more consistent internal standard.

Discriminative Power and System Rankings

Beyond reliability, an evaluation method must be able to distinguish between systems of similar quality. RankME proved to be more discriminative than Likert scales, which often resulted in statistical "ties."

System rankings produced by different evaluation methods. RankME provides clearer differentiation between systems compared to Likert scales.

For the "Quality" criterion, Likert scales failed to distinguish between the Slug2Slug and TGen systems. In contrast, RankME provided a clear, statistically significant ranking of all three systems. This increased sensitivity is crucial for fine-grained error analysis and model development.

Decoupling Criteria

The study also confirmed the existence of the "halo effect." When evaluated together (Setup 1), naturalness and informativeness scores were heavily correlated. By separating these tasks (Setup 2), the researchers successfully decoupled the criteria. This allows developers to identify specific weaknesses—for example, a system that produces very natural-sounding text but fails to include all required information.

Conclusion

RankME offers a more reliable and discriminative alternative to traditional NLG evaluation methods. By leveraging human strengths in relative comparison and providing continuous scales, it produces more consistent data for benchmarking AI systems. Furthermore, when combined with Bayesian estimation techniques like TrueSkill, RankME provides a cost-effective path toward stable and trustworthy system rankings.

Summary updated

Abstract

Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.
BibTeX
@inproceedings{novikova2018rankme,
  author = {Jekaterina Novikova and Ondrej Dusek and Verena Rieser},
  title = {{RankME: Reliable Human Ratings for Natural Language Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=SJWZ4mZuZr},
}
Figure