Improving Cross-Domain Hate Speech Generalizability with Emotion Knowledge

Shi Yin Hong, Susan Gauch

Abstract

Automated hate speech (HS) detection systems frequently struggle with cross-domain generalization, as models trained on one social media platform often fail to maintain performance on others due to varying linguistic styles and contexts. This study proposes a multitask learning framework that incorporates emotion knowledge (EK) to improve model robustness. By jointly training on HS detection and emotion classification using shared encoder parameters, the framework leverages consistent emotional signals—such as anger and disgust—as a bridge between domains. Experimental results across six diverse datasets demonstrate that this approach yields significant performance gains, particularly when using coarse-grained emotion categories, with improvements in cross-domain F1 scores reaching up to 18.1%.

Introduction

The detection of online hate speech is a critical yet challenging task. Current state-of-the-art models often overfit to the specific characteristics of their training data, such as the slang of Twitter or the discussion style of Reddit. Consequently, when these models are deployed in new environments, their accuracy drops significantly. This lack of generalizability stems from the dynamic nature of toxic language. However, while the specific words used to express hate may change, the underlying emotional intent remains relatively stable. This research explores whether "emotion knowledge" can serve as a universal feature set to help models identify hateful content regardless of the platform.

Methodology

The researchers implemented a multitask architecture designed to learn generalized representations. The model performs two concurrent tasks: primary binary hate speech detection and auxiliary emotion classification.

Architecture of the multitask framework for joint training of hate speech detection and emotion knowledge.

The architecture utilizes a shared encoder—either a standard BERT model or fBERT (pre-trained on offensive language)—to extract features. By sharing parameters, the emotion classification task acts as a regularizer, preventing the model from overfitting to domain-specific hate speech patterns. The study compared two emotion taxonomies:

  • $GE_{go}$: A fine-grained set of 28 emotions from the GoEmotions dataset.
  • $GE_{ek}$: A coarse-grained set of 6 basic emotions (Anger, Disgust, Fear, Joy, Sadness, Surprise) based on Ekman’s theory.

Experimental Setup and Datasets

To evaluate cross-domain performance, the study employed six publicly available hate speech datasets covering platforms like Twitter, Wikipedia, Facebook, and Reddit. The GoEmotions dataset provided the ground truth for the auxiliary emotion task. Evaluation followed a "leave-one-domain-out" protocol, where the model was trained on one dataset and tested on all others to measure its ability to generalize to unseen environments.

Results and Discussion

The inclusion of emotion knowledge consistently improved performance over baseline models trained solely on hate speech data.

The Impact of Emotion Granularity

A key finding was that the simpler 6-class emotion set ($GE_{ek}$) often outperformed the more complex 28-class set ($GE_{go}$) in terms of stability and generalization. Fine-grained emotions appeared to introduce noise or platform-specific sentiments that did not translate well across different social media contexts.

Cross-domain performance (Binary F1) and percentage improvements using different emotion corpora ($GE_{go}$ vs. $GE_{ek}$).

Performance Gains

The "emotion-enriched" models showed substantial improvements:

  • Maximum Improvement: Generalization performance increased by up to 18.1% in specific cross-platform transfers (e.g., Twitter to Facebook).
  • Average Gains: Across all experiments, the average F1 score saw an uplift of approximately 8.5%.
  • Base Model Sensitivity: While general-purpose BERT models saw the largest relative improvements, fBERT maintained higher absolute performance, suggesting that emotion knowledge and domain-specific pre-training are complementary strategies.

Conclusion and Future Work

Integrating emotion knowledge via multitask learning significantly enhances the cross-domain generalizability of hate speech detectors. By focusing on the emotional core of toxic language, models become less sensitive to platform-specific linguistic shifts. Future research should investigate the utility of this approach for detecting implicit hate speech and sarcasm, where emotional cues are often the primary indicators of intent. Additionally, expanding the study to include multi-class hate speech categories could provide further insights into how specific emotions correlate with different types of toxicity.

Summary updated

Abstract

Reliable automatic hate speech (HS) detection systems must adapt to the in-flow of diverse new data to curtail hate speech. However, hate speech detection systems commonly lack generalizability in identifying hate speech dissimilar to data used in training, impeding their robustness in real-world deployments. In this work, we propose a hate speech generalization framework that leverages emotion knowledge in a multitask architecture to improve the generalizability of hate speech detection in a cross-domain setting. We investigate emotion corpora with varying emotion categorical scopes to determine the best corpus scope for supplying emotion knowledge to foster generalized hate speech detection. We further assess the relationship between using pretrained Transformers models adapted for hate speech and its effect on our emotion-enriched hate speech generalization model. We perform extensive experiments on six publicly available datasets sourced from different online domains and show that our emotion-enriched HS detection generalization method demonstrates consistent generalization improvement in cross-domain evaluation, increasing generalization performance up to 18.1% and average cross-domain performance up to 8.5%, according to the F1 measure.
BibTeX
@inproceedings{hong2023improving,
  author = {Shi Yin Hong and Susan Gauch},
  title = {{Improving Cross-Domain Hate Speech Generalizability with Emotion Knowledge}},
  booktitle = {PACLIC},
  year = {2023},
  url = {https://openreview.net/forum?id=Hamk1vuAhT},
}
Figure