Insights into developing analytical categorization schemes: three problem types related to annotation agreement

Pihla Toivanen, Eetu Mäkelä, Antti KannerNAACL 2025

TL;DR

This paper identifies three common, yet often overlooked, challenges in developing annotation schemes for text analysis: negated presence, implicit presence, and low proportional presence of categories. By analyzing a case study of Finnish news articles on alcohol policy, the authors demonstrate how these issues lead to lower inter-annotator agreement and suggest practical recommendations for improving annotation scheme design.

What Problem Does This Paper Address?

In computational social sciences, coding text data into structured categories is crucial for building machine learning models that can analyze large datasets. This process, known as annotation, relies on well-defined schemes and consistent human coders. While factors like the number of annotators or categories are known to influence agreement, this paper highlights subtler, text-intrinsic problems that can significantly degrade annotation quality. Specifically, it addresses the challenge of developing robust annotation schemes that minimize disagreement, even when dealing with complex or nuanced textual content, which is essential for reliable automated text analysis.

Methods

The authors employed a qualitative approach, focusing on a case study of Finnish news media articles related to alcohol policy. They conducted an iterative process of close reading of both the articles and the annotations to identify patterns of disagreement. This led to the identification of three specific "problem types" related to how category-relevant content is presented in the text. To quantify the impact of these problem types, they then annotated a subset of articles for the presence of these features and compared agreement statistics, annotation times, and article lengths between articles exhibiting these features and those that did not.

Datasets

The study utilized a base dataset of 33,902 Finnish news media articles mentioning "alkoholi" (alcohol) and its word forms from four news outlets. For the analysis, three annotation rounds were conducted on randomly selected subsets of these articles. The first round involved a single-label scheme, while the second and third rounds used a multi-label scheme. A dataset of 1,500 articles with three annotations each was also created for training a text classifier, with the third annotation round contributing to this effort. No new benchmark datasets were introduced; rather, existing data was used to analyze annotation challenges.

Key Insights and Ideas

  • Implicit vs. Explicit: The way information is presented matters. Simply mentioning a concept isn't enough; its focality and context within the article are key.
  • Negation is Tricky: When categories are expressed through their opposites (e.g., "abstaining from alcohol for health benefits" instead of "alcohol harms"), annotators struggle to agree.
  • Low Presence, High Disagreement: If the relevant content is a small part of the article or not central to its main theme, annotators are more likely to disagree. This can be due to limited coverage or a lack of focal importance.
  • Interconnected Categories: When categories are conceptually linked (e.g., alcohol legislation and alcohol harms), it becomes difficult to assign an article to just one, leading to confusion.
  • Beyond Simple Agreement Metrics: While metrics like Cohen's Kappa are standard, they don't fully capture the reasons for disagreement. Understanding why annotators disagree is crucial for improving schemes.

Key Results

The study found that articles exhibiting the identified problem types (low proportional presence, implicit presence, or negated presence) consistently showed lower levels of annotation agreement compared to articles where these features were absent. Annotation times were also longer for articles with these challenging features, even when the article length was shorter, suggesting increased cognitive load for annotators. For instance, articles with implicit presence had a significantly lower proportion of complete agreement (4% vs. 53%) and longer annotation times. Similarly, articles with negated presence had zero complete agreement compared to 47% for directly present categories.

Limitations and Future Work

The authors acknowledge that their study is based on a single case study of alcohol policy media articles. However, they believe the identified problem types are likely to be relevant across various text types and annotation tasks. Future work could involve systematically testing these problem types across different domains and exploring more sophisticated methods for automatically detecting and mitigating these issues during the annotation process. They also suggest that future annotation scheme design should explicitly consider how to handle these nuances to improve inter-annotator agreement and the quality of the resulting data.

Summary updated

BibTeX
@inproceedings{toivanen2025insights,
  author = {Pihla Toivanen and Eetu Mäkelä and Antti Kanner},
  title = {{Insights into developing analytical categorization schemes: three problem types related to annotation agreement}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure