Introduction
Hate speech detection has traditionally focused on sentence-level classification, categorizing an entire input as either "hateful" or "neutral." However, this coarse-grained approach often fails to capture the nuances of offensive language in dynamic environments like social media or subcultural artistic expressions. This paper addresses the limitations of binary classification by shifting the focus to word-level (token) detection. Specifically, it tackles the challenges of code-mixing—where users blend languages like German and English—and the rapid evolution of neologisms, such as creative slurs and compounds that evade static lexicons.
Methodology: Token Classification
The researchers framed hate speech detection as a token classification task using XLM-R, a robust multilingual transformer model. Unlike sentence-level models, this approach assigns a specific tag to every word in a sequence:
- n: Non-hateful word.
- B-B: A single profane word or the start of a multi-word offensive sequence.
- B: The continuation of a multi-word profane sequence.
Three variations of the XLM-R model were evaluated:
- DFT (Domain Fine-Tuning): Fine-tuned exclusively on German data.
- DEFT (Domain-Enhanced Fine-Tuning): Fine-tuned on a bilingual mixture of German and English data.
- CMM (Code-Mixed Model): Fine-tuned on a smaller, specialized code-mixed dataset.
Datasets and the "Rap" Challenge
To evaluate these models, the authors developed five manually annotated word-level datasets spanning English (Reddit), German (Reddit/Sentiment), and a subset of Offensive Content Detection (OCD) data.
A primary contribution is the Rap dataset, consisting of German rap lyrics. This dataset serves as a rigorous "stress test" due to its heavy use of English loanwords as verbs (e.g., blowt) and complex German-English compounds. This domain highlights the model's ability to handle intra-word code-switching and creative profanity.
Key Results and Insights
The experimental results demonstrate that bilingual training significantly enhances a model's ability to generalize to unseen, code-mixed domains. Surprisingly, the DEFT model (bilingual) outperformed the CMM (code-mixed), suggesting that the volume and quality of monolingual data provide a stronger foundation for multilingual transfer than smaller, noisier code-mixed sets.
As shown in the results table, DEFT achieved the highest F1 scores on both the Rap (82.67) and Held-Out (81.43) datasets. When compared to Large Language Models (LLMs) like GPT-4o and Mistral, the fine-tuned token classifiers showed superior consistency. While LLMs often achieved high precision, they struggled with recall, frequently failing to identify the continuation tokens ('B' tags) in multi-word sequences.
Furthermore, the models exhibited a remarkable ability to detect neologisms. By leveraging subword-level patterns learned during multilingual pre-training, the models could identify brand-new slurs (e.g., Muschibattle) that were not present in the training data.
Limitations and Future Directions
While promising, the study notes several areas for improvement. The current datasets are relatively small, making models sensitive to training initialization. Additionally, the focus remains on explicit profanity; detecting implicit hate speech—such as sarcasm or "dog whistles"—at the word level remains a significant challenge. Future work may explore integrating explicit language tagging or expanding the methodology to other language pairs like Spanish-English or Hindi-English.
Conclusion
By moving to word-level detection and utilizing bilingual fine-tuning, this research provides a more granular and effective way to monitor offensive language in multilingual contexts. The DEFT model's success in the challenging domain of German rap lyrics underscores the power of multilingual domain transfer for capturing the evolving nature of digital hate speech.
Summary updated