Harmful Language Datasets: An Assessment of Robustness

Katerina Korre, John Pavlopoulos, Jeffrey Sorensen, Léo Laugier, Ion Androutsopoulos, Lucas Dixon, Alberto Barrón‐Cedeño2023 · DOI

TL;DR

This paper investigates the robustness of harmful language detection models by re-annotating existing datasets with different definitions. They found that inter-annotator agreement is consistently low, highlighting the subjective nature of classifying harmful language. Surprisingly, models trained on one dataset did not necessarily perform better on data from the same source when evaluated with new annotations, challenging previous assumptions about dataset distribution.

What Problem Does This Paper Address?

The online world is increasingly grappling with harmful language, from hate speech to general offensiveness. While automated detection systems are crucial for moderating content on social media, their effectiveness is hampered by several challenges. One major hurdle is the inherent subjectivity of what constitutes "harmful language." Researchers often use different definitions, leading to ambiguity and making it difficult to compare results or build truly generalizable models. This paper tackles this by systematically examining how different definitions impact annotation consistency and how well existing models perform when faced with data that has been re-annotated under these varied definitions.

Methods

The core of this research involves a two-pronged approach:

  1. Re-annotation Experiments: The authors took existing harmful language datasets and had them re-annotated by crowdworkers using five different definitions of harmful language (Toxicity, Hate Speech, Offensiveness, Abusiveness, and HOTA - Harmful, Offensive, Toxic, Abusive). This was done in multiple rounds to assess robustness.
  2. Model Evaluation: They then trained BERT-based classifiers on the original datasets and evaluated their performance on both the original annotations and the newly generated re-annotations. This allowed them to assess how well models generalize across different definitions and data sources.

Datasets

The study utilized four publicly available datasets:

  • Davidson (2017): This dataset was split into two subsets: DavidsonHS (for hate speech) and DavidsonOFF (for offensiveness).
  • TRAC-1 (Kumar et al., 2018b): This dataset focuses on aggression.
  • Toxkaggle (Jigsaw, 2019): This dataset is for toxic comment classification.

The authors created new, balanced datasets for their annotation experiments, ensuring an equal distribution of positive and negative instances and maintaining class balance within each batch.

Key Insights and Ideas

  • Subjectivity Reigns Supreme: Despite using clear definitions, inter-annotator agreement remained consistently low across all definitions and datasets. This strongly suggests that classifying harmful language is an inherently subjective task, even for human annotators.
  • Broader Definitions Might Be Better: Toxicity and HOTA (a broader, encompassing definition) showed slightly more consistent annotations, hinting that more general terms might lead to more reliable labeling.
  • Challenging the "Same Source" Assumption: Contrary to some prior research, the study found that models trained on a specific dataset did not consistently perform better when tested on re-annotated data from the same source. In fact, performance sometimes dropped, suggesting models struggle to generalize even within the same data distribution when definitions shift.
  • Cross-Dataset Generalization is Tricky: While models didn't always generalize well to data from the same source, there were instances where performance improved when testing on re-annotations from a different source, indicating a complex interplay between data source, definition, and model performance.

Key Results

  • Low Inter-Annotator Agreement: Krippendorf's alpha values, a measure of agreement, were consistently below 0.5 for most definitions and datasets, indicating significant disagreement among annotators.
  • Model Performance Fluctuations: When evaluating models on re-annotated data, accuracy often decreased compared to evaluation on original annotations. This was particularly true when testing on data from the same source as the training set.
  • Toxicity and HOTA Showed Relative Consistency: While still low, the annotations for "Toxicity" and "HOTA" demonstrated slightly higher consistency, suggesting these broader definitions might be more robust.
  • No Clear "Same Source" Advantage: The study found that models trained on DavidsonHS performed well on DavidsonHS test sets with original annotations (accuracy > 0.75). However, when tested on re-annotated DavidsonHS data, performance varied, and the expected advantage of same-source testing was not consistently observed.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Dataset Scope: Not all relevant toxicity datasets were studied, and only BERT-based classifiers were used.
  • Crowdsourcing Bias: The annotation relied solely on crowdsourcing. Future work could explore the impact of expert annotators and compare their agreement with crowdworkers.
  • Future Directions: The study suggests further research into more datasets and algorithms. It also highlights the importance of carefully considering the "DOs and DON'Ts" for annotation and dataset creation, emphasizing the need for robust annotation procedures to build more reliable harmful language detection models. The authors also point to the difficulty of detecting "implicit aggressiveness," which poses a challenge for both models and human annotators.

Summary updated

Abstract

The automated detection of harmful language has been of great importance for the online world, especially with the growing importance of social media and, consequently, polarisation. There are many open challenges to high quality detection of harmful text, from dataset creation to generalisable application, thus calling for more systematic studies. In this paper, we explore re-annotation as a means of examining the robustness of already existing labelled datasets, showing that, despite using alternative definitions, the inter-annotator agreement remains very inconsistent, highlighting the intrinsically subjective and variable nature of the task. In addition, we build automatic toxicity detectors using the existing datasets, with their original labels, and we evaluate them on our multi-definition and multi-source datasets. Surprisingly, while other studies show that hate speech detection models perform better on data that are derived from the same distribution as the training set, our analysis demonstrates this is not necessarily true.
BibTeX
@misc{korre2023harmful,
  author = {Katerina Korre and John Pavlopoulos and Jeffrey Sorensen and Léo Laugier and Ion Androutsopoulos and Lucas Dixon and Alberto Barrón‐Cedeño},
  title = {{Harmful Language Datasets: An Assessment of Robustness}},
  year = {2023},
  doi = {10.18653/v1/2023.woah-1.24},
}
Figure