LMU at HaSpeeDe3: Multi-Dataset Training for Cross-Domain Hate Speech Detection

Viktor Hangya, Alexander FraserEVALITA 2023 · OpenReview

Abstract

Cross-domain hate speech detection remains a significant challenge in natural language processing, particularly when target domains lack training data. This paper presents the LMU Munich team's approach to the HaSpeeDe3 shared task at EVALITA 2023. By leveraging prompt-based multitask learning and ensembling monolingual and multilingual models, our system achieved second place in both the political and zero-shot religious domains for Italian hate speech detection.

Introduction

Hate speech classifiers often struggle with domain shifts, where a model trained on general social media data fails when applied to specific, polarized topics like politics or religion. The HaSpeeDe3 shared task addressed this by challenging researchers to build robust detectors for Italian tweets. A primary difficulty was the religious track, which provided zero training data, requiring effective zero-shot transfer from external datasets.

Methodology

Our approach utilizes a two-step transfer learning pipeline: pre-finetuning on a mixture of external datasets followed by fine-tuning on the target domain (or zero-shot application).

Prompt-Based Multitask Learning

Instead of standard classification heads, we employ Prompt-Training using Pattern-Verbalizer-Pairs (PVPs). This frames classification as a Masked Language Modeling (MLM) task. For instance, a tweet is transformed into a prompt like: "Is this hate speech? [tweet]. [MASK]". The model then predicts the masked word (e.g., "Yes" for hate or "No" for non-hate).

Patterns and Verbalizers used for prompt-based training across different datasets.

This method is particularly powerful for multitask learning across datasets with mismatched taxonomies. It allows us to unify diverse labels—such as binary hate speech, fine-grained misogyny, and stereotypes—under a single vocabulary-prediction objective without complex multi-head architectures.

Experimental Setup

Datasets

  • Target (HaSpeeDe3): PolicyCorpusXL (5,600 political tweets) and ReligiousHate (3,000 religious tweets for testing only).
  • External Source Data: We experimented with three groupings: HaSpeeDe (past shared task data), It (adding Italian misogyny and immigrant-targeted hate speech), and Mixed (adding English political hate speech data).

Models

We evaluated several Pre-trained Language Models (PLMs), including Italian-specific models like AlBERTo and UmBERTo, alongside multilingual models like mBERT and XLM-R. Final predictions were generated using ensembles to maximize robustness.

Results and Discussion

Our system secured 2nd place in both task tracks. In the political domain, the monolingual ensemble achieved a Macro F1 of 91.44%. For the zero-shot religious domain, a mixed ensemble of monolingual and multilingual models reached a Macro F1 of 64.61%.

Performance comparison of individual PLMs and ensembles across political and religious domains.

Key Insights

  • Prompting as a Unifier: PVPs effectively pooled datasets with disparate labeling schemes, allowing the model to learn general representations of abusive language.
  • Language Specificity: Italian-specific models (AlBERTo) generally outperformed multilingual ones, suggesting that domain-matched pre-training on Italian Twitter data is highly valuable.
  • Ensembling for Robustness: While monolingual models were strong individually, ensembling them with multilingual models provided the best performance in the difficult zero-shot religious domain.

Conclusion

The LMU team's success at HaSpeeDe3 demonstrates the efficacy of prompt-based multitask learning for cross-domain transfer. By unifying diverse datasets through a shared MLM objective and leveraging domain-specific pre-trained models, we established a robust framework for detecting hate speech in highly polarized and low-resource contexts.

Summary updated

BibTeX
@inproceedings{hangya2023lmu,
  author = {Viktor Hangya and Alexander Fraser},
  title = {{LMU at HaSpeeDe3: Multi-Dataset Training for Cross-Domain Hate Speech Detection}},
  booktitle = {EVALITA},
  year = {2023},
  url = {https://openreview.net/forum?id=oJd9rZ9biU},
}
Figure