HATE-ITA: Hate Speech Detection in Italian Social Media Text

Debora Nozza, Federico Bianchi, Giuseppe AttanasioNAACL 2022

TL;DR

This paper introduces HATE-ITA, a novel resource for Italian hate speech detection, by leveraging multilingual models trained on a combined English and Italian dataset. This approach significantly outperforms monolingual models, establishing a new benchmark for Italian hate speech detection and encouraging research in low-resource languages.

What Problem Does This Paper Address?

Online hate speech is a pervasive and harmful phenomenon that requires effective detection mechanisms. While Natural Language Processing (NLP) has made strides in this area, most research has focused on English, leaving a significant gap in the detection capabilities for other languages, particularly those with fewer available resources like Italian. Existing Italian hate speech datasets are often limited in scope and size, and a robust, competitive baseline for detection has been lacking. This paper aims to bridge this gap by developing and releasing a set of high-performing models for Italian hate speech detection.

Methods

The core of this work lies in the application of multilingual language models for hate speech detection in Italian. Instead of relying solely on limited Italian data, the authors combine a large English hate speech dataset with a smaller Italian dataset to train models. This approach leverages the idea that knowledge learned from a high-resource language (English) can be transferred to a low-resource language (Italian), a strategy known as cross-lingual transfer learning.

Specifically, they experiment with several pre-trained multilingual models, including XLM-Roberta (Base and Large), mBERT, and XLM-Twitter. These models are then fine-tuned under three experimental frameworks:

  1. Mono-lingual (MONO): Training exclusively on Italian data.
  2. Multi-lingual (MULTI): Training on a combined English and Italian dataset.
  3. Zero-shot (ZERO): Training only on English data and testing on Italian data.

The models are evaluated on established Italian test sets, and their performance is compared against monolingual models and a previously developed Italian hate speech detection model (DeHateBert).

Datasets

The paper utilizes a combination of existing datasets for training and evaluation:

  • Italian Datasets:
    • Evalita shared tasks corpora: AMI18 (misogyny identification) and HaSpeeDe18 (hate speech towards immigrants).
  • These datasets are used for training, validation, and testing, with specific splits provided by the shared tasks.
  • English Datasets:
    • MlMa (multi-lingual multi-aspect hate speech analysis).
  • ETHOS (multi-label English hate speech detection from Reddit).
  • A large dataset collected from YouTube, Twitter, and Reddit by Kennedy et al. (2020c).
  • English posts from Twitter and Gab by Mathew et al. (2021).
  • Gab Hate Corpus (GHC) by Kennedy et al. (2020a).
  • A dynamically created benchmark corpus by Kiela et al. (2021).

The English datasets are combined to form a large training set for the multilingual models. The paper also uses the Multi-Lingual HateCheck (MHC) dataset as an external validation set to test the models' robustness across various hate speech functionalities.

Key Insights and Ideas

  • Multilingual Models are Key for Low-Resource Languages: The most significant insight is that multilingual models, when trained on a combination of high-resource (English) and low-resource (Italian) data, significantly outperform monolingual models. This demonstrates a powerful strategy for addressing data scarcity in NLP tasks for under-resourced languages.
  • Cross-Lingual Transfer is Effective: The "zero-shot" experiments, where models trained only on English data are tested on Italian, show promising results, indicating that knowledge transfer is indeed happening. However, combining data (MULTI) yields superior performance, suggesting that even a small amount of target-language data is highly beneficial.
  • HATE-ITA as a Valuable Benchmark: The release of HATE-ITA, a set of fine-tuned multilingual models and associated APIs, provides a much-needed benchmark for Italian hate speech detection, encouraging further research and development in this area.
  • Adaptability to Language-Specific Slurs: The models show an ability to adapt well to language-specific slurs, which is crucial for effective hate speech detection in any given language.

Key Results

  • Superior Performance: The best multilingual model (XLM-Large) achieved significantly better performance (Macro-F1 scores) compared to the best monolingual model. For instance, on the combined Italian test sets, multilingual models consistently outperformed monolingual ones.
  • Benchmark Performance: The HATE-ITA models set a new baseline on two state-of-the-art Italian hate speech detection datasets.
  • Robustness: The models demonstrated consistent performance across different Italian datasets and on the Multi-Lingual HateCheck, indicating robustness. XLM-Twitter and XLM-Large, in particular, outperformed existing baselines on MHC.
  • Qualitative Analysis: Using SHAP (SHapley Additive exPlanations), the authors provided qualitative insights into model predictions, highlighting how the models interpret complex Italian expressions, including those with ambiguous meanings or culturally specific taboo phrases.

Limitations and Future Work

  • Limited Italian Data: While the multilingual approach is effective, the authors acknowledge that the amount of Italian data used is still relatively small. They suggest that further increasing the Italian data could lead to even better performance.
  • Uncertainty in Precision: The authors express caution, stating that the results should not be interpreted as a definitive assessment of hate speech detection in Italian. They are unsure if the model can maintain stable and fair precision across all targets and categories, and ethical considerations regarding potential overlooked nuances are highlighted.
  • Future Directions: The paper suggests that HATE-ITA can be used to identify areas and targets that are still not well-covered, thus guiding future research. Further benchmarking is needed to fully understand the model's capabilities across different targets.

Summary updated

BibTeX
@inproceedings{nozza2022hate,
  author = {Debora Nozza and Federico Bianchi and Giuseppe Attanasio},
  title = {{HATE-ITA: Hate Speech Detection in Italian Social Media Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure