Rethinking the Agreement in Human Evaluation Tasks

Jacopo Amidei, Paul Piwek, Alistair WillisCOLING 2018

TL;DR

This paper argues that high inter-annotator agreement (IAA) in natural language generation (NLG) tasks, particularly Automatic Question Generation (AQG), might be counterproductive. The authors propose that the inherent variability and richness of human language lead to unavoidable disagreements among annotators, and striving for perfect agreement could stifle the very qualities NLG systems aim to capture. They suggest rethinking how IAA is interpreted and propose new directions for evaluation.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), high inter-annotator agreement (IAA) is generally considered a hallmark of reliable data and annotation quality. It's used to assess data reliability, identify annotation issues, and set performance benchmarks. However, this paper questions the universal desirability of high IAA, especially in Natural Language Generation (NLG) tasks. NLG tasks, like generating questions from text, involve subjective judgments about quality, which can be influenced by a multitude of factors. The authors contend that an overemphasis on reducing annotator disagreement in NLG might inadvertently lead systems to produce outputs that are less natural and diverse, moving away from the goal of generating human-like language. They specifically focus on the domain of Automatic Question Generation (AQG) to explore this issue.

Methods

The authors conducted experiments involving human annotation of automatically generated questions. They developed and refined annotation guidelines through several iterations of discussions and pilot evaluations. The evaluation was structured along two dimensions: a linguistic one (grammaticality, comprehensibility, fluency) and a task-oriented one (pertinence). They analyzed the sources of disagreement among annotators, categorizing them into five main areas: style and taste, background knowledge, personal assumptions, use of common sense inferences, and attention to detail. The core of their methodological contribution lies in the analysis of these divergences and the subsequent argument for rethinking IAA metrics.

Datasets

The experiments utilized paragraphs and questions from the SQuAD dataset. Some questions were human-generated from the SQUAD dataset, while others were synthetically modified or generated by the authors. One question was generated using a publicly available question generation algorithm.

Key Insights and Ideas

  • High IAA is not always desirable in NLG: The paper challenges the conventional wisdom that high IAA is the ultimate goal in NLG evaluation.
  • Language variability is key: The inherent richness, variability, and beauty of human languages naturally lead to disagreements among annotators, which shouldn't be seen as a problem to be eliminated.
  • Over-constraining guidelines can be detrimental: Forcing annotators into strict agreement can artificially limit their interpretation and stifle the generation of diverse, human-like language.
  • Sources of disagreement are multifaceted: Disagreements stem from subjective factors like personal taste, background knowledge, assumptions, common sense, and attention to detail, which are deeply intertwined with how humans understand and use language.
  • Kappa coefficients might be inadequate: The paper suggests that standard IAA metrics like Kappa might not be suitable for NLG tasks due to their inability to account for the nuanced nature of language evaluation.

Key Results

The experiments consistently yielded low IAA scores, with the lowest score being 0.11. These low scores, when interpreted through standard Kappa scales, would typically be considered inadequate. However, the authors argue that these low scores are indicative of the task's nature and the inherent variability of human language, rather than a failure of the annotation process. They identified specific examples of how annotators diverged based on the five categories of disagreement mentioned in the Methods section.

Limitations and Future Work

The authors acknowledge that their study focused on the text2text AQG task and that the findings might need further validation across other NLG tasks. They propose several future research directions:

  • Reconsidering IAA interpretation: The scales for Kappa interpretation should be re-evaluated for NLG tasks.
  • Introducing confidence intervals: Instead of a single agreement score, confidence intervals could be used to define acceptable ranges for IAA, acknowledging that some level of disagreement is natural.
  • Focusing on internal consistency: The consistency of an individual annotator's judgments should be considered alongside inter-annotator agreement.
  • Developing shared evaluation guidelines and corpora: For specific NLG tasks, common evaluation guidelines and datasets could foster more uniform evaluation practices.
  • Analyzing the correlation between IAA and linguistic difficulty: Understanding how linguistic complexity affects IAA could help in setting task-specific thresholds.

Summary updated

BibTeX
@inproceedings{amidei2018rethinking,
  author = {Jacopo Amidei and Paul Piwek and Alistair Willis},
  title = {{Rethinking the Agreement in Human Evaluation Tasks}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure