RankME: Reliable Human Ratings for Natural Language Generation

Jekaterina Novikova, Ondrej Dusek, Verena Rieser

Abstract

Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.

Introduction

Human evaluation is the gold standard for Natural Language Generation (NLG), but it is often plagued by inconsistency and bias. Researchers typically face three major hurdles: rater inconsistency, "criteria bleed" (where distinct metrics like naturalness and informativeness are conflated), and the limitations of discrete scales (like 5-point Likert scales) that fail to capture nuanced differences between modern systems. This paper argues that these are not just issues of user preference, but flaws in experimental design that can be addressed through better methodology.

The RankME Method

We propose RankME (Rank-based Magnitude Estimation), a new evaluation method that merges two powerful concepts: Magnitude Estimation (ME) and Relative Ranking. In ME, raters compare a target sentence to a reference benchmark on a continuous scale without fixed endpoints. RankME extends this by asking subjects to provide a relative ranking of all target sentences in a set, rather than assessing them in isolation.

Comparison of evaluation methods across Direct Assessment (DA), Relative Ranking (RR), Discrete Scale (DS), and Continuous Scale (CS).

As shown in the table above, RankME is unique in its combination of relative ranking and continuous scales, distinguishing it from traditional Likert scales and Plain ME.

Experimental Design

To validate RankME, we used outputs from three state-of-the-art systems (Sheffield NLP, TGen, and Slug2Slug) and compared it against Likert scales and Plain ME across two setups:

  • Setup 1: Raters evaluated all criteria (naturalness, informativeness, and overall quality) in a single task.
  • Setup 2: Raters evaluated criteria in separate, individual tasks to prevent one score from "bleeding" into another.

To make the ranking process more efficient, we paired RankME with TrueSkill, a Bayesian rating system originally designed for competitive matchmaking, to estimate system quality from pairwise comparisons.

Key Results

Higher Reliability and Discrimination

RankME achieved the highest Intra-class Correlation Coefficient (ICC), a key measure of agreement between raters. For naturalness, RankME reached an ICC of 0.42, compared to just 0.27 for Plain ME. Furthermore, RankME provided more distinctive system rankings, clearly separating system performance where Likert scales often resulted in overlapping confidence intervals.

Mitigating Criteria Bleed

A striking finding was that separating evaluation tasks (Setup 2) drastically reduced the correlation between distinct criteria. When judges were not asked to look at everything at once, they were much better at spotting systems that were fluent but factually incorrect.

Spearman's rho correlations between criteria across different evaluation methods and setups.

As illustrated in the correlation tables above, the relationship between "quality" and specific criteria like "naturalness" changes significantly between setups. In Setup 1, raters often used naturalness as a proxy for quality (correlation of 0.54 for Likert), but this correlation disappears in Setup 2, indicating that judges were better able to isolate specific flaws when tasks were separated.

Conclusion

RankME offers a more reliable and discriminative alternative to traditional NLG evaluation. By utilizing continuous scales and relative rankings, and by strategically separating evaluation tasks, researchers can obtain higher-quality human judgments. When combined with Bayesian estimation techniques like TrueSkill, RankME also proves to be a cost-effective solution for large-scale system benchmarking.

Summary updated

Abstract

Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.
BibTeX
@inproceedings{novikova2018rankme,
  author = {Jekaterina Novikova and Ondrej Dusek and Verena Rieser},
  title = {{RankME: Reliable Human Ratings for Natural Language Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=SJWZ4mZuZr},
}
Figure