Improving Hate Speech Classification with Cross-Taxonomy Dataset Integration

Jan Fillies, Adrian PaschkeNAACL 2025

TL;DR

This paper tackles the challenge of hate speech detection by proposing a novel approach to integrate diverse hate speech definitions and datasets into a single, unified model. By creating a comprehensive general taxonomy and employing federated learning, they demonstrate improved classification performance and broader applicability across different contexts.

What Problem Does This Paper Address?

The field of hate speech detection is plagued by a significant challenge: the lack of a universally agreed-upon definition of hate speech. Different social media platforms, legal frameworks, and research communities employ their own distinct, yet often overlapping, definitions. This fragmentation leads to a proliferation of specialized datasets, each annotated according to its own schema. Consequently, training a single, robust hate speech classifier that can generalize across these diverse definitions is incredibly difficult. Comparing and merging existing datasets becomes a complex task, hindering progress and limiting the effectiveness of automated hate speech detection systems.

Methods

The core of this research lies in developing a unified framework for hate speech classification. The methodology involves several key steps:

  • General Taxonomy Creation: The authors merge existing hate speech taxonomies into a single, comprehensive taxonomy. This involves identifying shared classes, resolving discrepancies, and establishing hierarchical relationships between different categories of hate speech.
  • Cross-Taxonomy Annotation: One of the existing datasets is re-annotated to conform to the newly created general taxonomy. This process involves mapping labels from the original annotation schema to the unified one, addressing potential mismatches and missing labels.
  • Multi-Label Classification Model: A pre-trained transformer-based model (specifically, RoBERTa) is fine-tuned as a multi-label classifier. This model is trained on the re-annotated dataset.
  • Iterative Fine-tuning and Evaluation: The trained classifier is then used to re-annotate a second, distinct hate speech dataset. The human annotations from this second dataset are then merged with the model's predictions to create a hybrid dataset. This hybrid dataset is used to further fine-tune the model. This iterative process allows the model to learn from diverse definitions and improve its performance.
  • Federated Learning Principles: While not explicitly implementing a full federated learning system, the approach aligns with its principles by aiming to train a model on diverse data sources without necessarily sharing the raw, sensitive datasets. The focus is on exchanging learned patterns and models.

Datasets

The research utilizes several datasets:

  • Vidgen et al. (2021): A large, dynamically generated dataset of 41,255 entries, comprising 11 English-language training datasets. It features a hierarchical taxonomy and classifies entries into hate and non-hate categories, with hate further divided into five types.
  • Fanton et al. (2021): A human-in-the-loop dataset of 5,000 hateful statements, with annotations for 10 specific target groups.
  • Fillies et al. (2023b): A dataset of 88,395 Discord messages used to provide non-hateful statements, as the primary datasets were heavily skewed towards hateful content.
  • Ljubešić et al. (2021): A YouTube comments dataset used for validation and independent evaluation of the classifier's performance.

Key Insights and Ideas

  • Unified Taxonomy is Key: The core idea is that a single, well-structured taxonomy can bridge the gap between disparate hate speech definitions, enabling a unified classification approach.
  • Iterative Refinement Works: Fine-tuning a model on one dataset and then using its predictions to augment another dataset for further training significantly improves performance and generalization.
  • Semantic Relationships Matter: Encoding hierarchical relationships within the taxonomy (e.g., "skin_color" as a parent of "black") helps the model understand nuanced definitions and improve classification.
  • Federated Learning Potential: The approach lays the groundwork for federated hate speech detection, where models can be trained on distributed data without direct data sharing, addressing privacy concerns.

Key Results

  • The proposed approach, after iterative fine-tuning, led to improved classification performance. For instance, the F1-Score on an independent evaluation test set increased from 0.73 to 0.84.
  • The accuracy also saw a significant increase after adjusting the taxonomy, demonstrating the effectiveness of the unified approach.
  • The research successfully demonstrated that a general taxonomy can be created and used for multi-label hate speech classification, integrating diverse datasets and definitions.

Limitations and Future Work

  • Not a Universal Proof: The authors acknowledge that their work doesn't prove that all datasets and taxonomies can be perfectly merged. Some subparts of the example taxonomies could not be merged.
  • Initial Mislabeling: The first round of training involves potentially mislabeled data, which could lead to underperformance. Algorithmically generated annotations might also propagate biases.
  • Bias Mitigation and Quality Assurance: Further work is needed on bias mitigation and quality assurance in hate speech detection.
  • Ontology Matching: Deeper issues, like the misclassification of "LGBTQ+" terms, might require more advanced solutions, potentially utilizing ontology matching techniques in the future.
  • Broader Taxonomy Validation: Future research should explore automatic matching of taxonomies on both logical and semantic levels, and potentially create a hate speech ontology.

Summary updated

BibTeX
@inproceedings{fillies2025improving,
  author = {Jan Fillies and Adrian Paschke},
  title = {{Improving Hate Speech Classification with Cross-Taxonomy Dataset Integration}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure