k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations

Ka Wong, Praveen K. ParitoshACL 2022 · DOI

In Natural Language Processing (NLP), crowdsourcing is the standard method for generating "ground truth" labels. Because individual human judgments are often noisy or subjective, researchers typically collect multiple ratings per item and use their aggregate—such as the mean or majority vote—as the final label. However, a fundamental mismatch exists in how the quality of this data is reported. While models consume aggregated data, researchers continue to report Inter-Rater Reliability (IRR), which measures the consistency of individual ratings.

This paper, presented at ACL 2022, argues that IRR is the wrong unit of analysis for aggregated data. Instead, researchers should report k-Rater Reliability (kRR), a metric that correctly reflects the stability and reproducibility of the data actually used in downstream tasks.

The Problem: The Wrong Unit of Analysis

Reporting IRR for aggregated data leads to an under-reporting of dataset quality. If a dataset has a moderate IRR (e.g., 0.60), it is often stigmatized as "noisy" or unreliable. Yet, the "wisdom of the crowd" suggests that the average of these ratings is far more stable than any single rater. By failing to account for the aggregation process, the NLP community may be dismissing valid research on subjective tasks—like sentiment analysis or toxicity detection—due to a perceived lack of reliability that is actually a measurement error.

Methodology: Measuring kRR

The authors propose and validate four distinct methods for calculating kRR:

  1. Empirical Approach: Conducting two independent replications of the entire data collection process and measuring the agreement between the two sets of aggregated scores.
  2. Analytical Approach (ICC): Utilizing Intraclass Correlation ($ICC(k)$), a statistical method designed for continuous data where the mean is the unit of aggregation.
  3. Bootstrap Approach: A resampling technique that simulates replications by sampling from the existing pool of ratings with replacement, offering a cost-effective alternative to full replication.
  4. Spearman-Brown Prophecy Formula: A psychometric tool that predicts the reliability of a $k$-rater aggregate based on the reliability of a smaller subset (e.g., two raters).
Figure 1: Comparison of kRR calculation methods showing how reliability scales with the number of raters (k).

As shown in Figure 1, these methods converge significantly as the number of raters ($k$) increases. The empirical, analytical, and bootstrap methods yield nearly identical results, suggesting that researchers can confidently use simpler resampling or statistical methods without the need for expensive full-scale replications.

Case Study: WordSim-353

To demonstrate the impact of kRR, the authors analyzed the WordSim-353 benchmark, a dataset for word similarity containing 353 word pairs rated by 13 workers. They conducted two new replications on Amazon Mechanical Turk to validate their findings against the original 2001 data.

The results highlight a dramatic difference between individual and aggregated reliability. While individual ratings (IRR) hovered between 0.574 and 0.590—scores typically considered mediocre—the aggregated reliability for 13 raters (kRR) reached as high as 0.953.

Figure 2: Reliability results for WordSim-353, comparing single-rating IRR to 13-rating kRR.

Conclusion and Future Directions

The shift from IRR to kRR reveals that many NLP datasets are far more robust than previously thought. This finding is particularly important for subjective domains where high individual agreement is naturally difficult to achieve.

However, limitations remain. Current analytical methods like ICC are best suited for continuous data (means). Future work is needed to establish standard kRR metrics for categorical data (majority voting) and to develop new "rules of thumb" for what constitutes a "good" kRR score, as the thresholds for aggregated data should naturally be higher than those for individuals. By adopting kRR, the NLP community can move toward a more accurate and fair assessment of crowdsourced data quality.

Summary updated

BibTeX
@inproceedings{wong2022k,
  author = {Ka Wong and Praveen K. Paritosh},
  title = {{k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.42},
}
Figure