TL;DR
This paper investigates the effectiveness of zero-shot, cross-lingual transfer learning for hate speech detection, revealing significant limitations. The authors demonstrate that current models struggle to generalize due to language-specific taboo interjections being misinterpreted as hate speech, highlighting the need for more nuanced, language-aware approaches.
What Problem Does This Paper Address?
Hate speech on social media is a growing concern, and automatic detection methods are crucial. However, most existing research and labeled datasets are English-centric. While zero-shot, cross-lingual learning (training a model on one language and applying it to another unseen language) seems like a promising solution to bridge this gap, its effectiveness and limitations in the context of hate speech detection remain underexplored. This paper specifically addresses the challenge of detecting hate speech in languages other than English, particularly towards immigrants and women, by examining the performance and underlying reasons for failure in zero-shot, cross-lingual setups.
Methods
The core of this research involves fine-tuning multilingual BERT (mBERT) and XLM-R models for hate speech detection. The authors explore several learning settings, including:
- Zero-shot, cross-lingual: Training on one language and testing on an unseen language.
- Monolingual: Training and testing on the same language.
- Few-shot, cross-lingual: Training on one language with a small percentage of samples from the target language.
- Augmented cross-lingual: Training on multiple languages and testing on one of those languages.
To understand why models make certain predictions, they employ the Sampling and Occlusion (SOC) algorithm for post-hoc explanations. SOC assigns importance scores to words or phrases, revealing how they contribute to the model's classification. This allows for a hierarchical understanding of compositional semantics and how the model interprets language.
Datasets
The study utilizes benchmark datasets for hate speech detection, specifically focusing on hate speech targeting immigrants and women.
- English and Spanish: Data from the HatEval shared task (Basile et al., 2019).
- Italian: Data from two sources:
- The automatic misogyny identification challenge (AMI) for hate speech towards women (Fersini et al., 2018).
- The hate speech detection shared task on Facebook and Twitter (HaSpeeDe) for hate speech towards immigrants (Bosco et al., 2018).
The paper provides a table detailing the splits of these datasets across training, development, and testing sets for each language and target type.
Key Insights and Ideas
- Taboo interjections are a major hurdle: The most significant finding is that common, language-specific taboo interjections (e.g., "porca puttana," "puta") are frequently misinterpreted by zero-shot, cross-lingual models. These words, while potentially offensive in a literal sense, are often used as intensifiers or exclamations in non-hateful contexts in languages like Italian and Spanish.
- Literal meaning over contextual use: Zero-shot models, trained primarily on English data where such interjections are less common or used differently, tend to treat these words literally. This leads them to incorrectly classify non-hateful sentences containing these expressions as hate speech, particularly misogynous hate speech.
- English data can be over-sensitive: The English dataset, due to its specific data collection methods (e.g., using hashtags like #SendThemBack), can lead to overfitting and poor performance when models are trained solely on it.
- Target-specific models are necessary: The paper suggests that treating hate speech towards different targets (immigrants vs. women) requires separate models. Merging data for both targets in a zero-shot, cross-lingual setting results in compromised performance, averaging out the strengths and weaknesses.
- XLM-R also struggles: Even with XLM-R, a more powerful cross-lingual model, the performance drop in zero-shot, cross-lingual settings for hate speech detection, particularly for women, is substantial, reinforcing the impact of language-specific nuances.
Key Results
- Zero-shot, cross-lingual performance varies by target: For hate speech towards immigrants, zero-shot, cross-lingual models show relatively good performance, especially when testing on Italian and Spanish. However, for hate speech towards women, performance drops significantly.
- Misclassification of taboo interjections: In Italian and Spanish test sets, a high percentage of instances (29% and 78% respectively) contain taboo interjections. Zero-shot models correctly predict only a small fraction of these (3% and 58%), while monolingual models achieve much higher accuracy (99% and 82%).
- Example of misclassification: Figure 1 illustrates how a non-hateful Italian tweet containing "porca puttana" is misclassified by a zero-shot, cross-lingual model but correctly classified by a monolingual Italian model.
- Impact on XLM-R: Zero-shot, cross-lingual settings with XLM-R show a significant drop in macro-F1 scores (e.g., 36% decrease) compared to monolingual settings, particularly when testing on Italian and Spanish.
Limitations and Future Work
- Limitations: The paper explicitly states that zero-shot, cross-lingual models, in their current form, are not a viable solution for hate speech detection in non-English languages due to their inability to capture language-specific nuances like taboo interjections. The findings are specific to the considered languages (English, Italian, Spanish) and data sets.
- Future Work: The authors plan to:
- Expand the evaluation to other languages.
- Investigate solutions based on bias mitigation techniques.
- Explore pragmatic role-aware models to better understand context.
- Model language's social factors, such as speaker and receiver characteristics, to improve hate speech detection classifiers.
Summary updated