Evaluating the Effectiveness of Natural Language Inference for Hate Speech Detection in Languages with Limited Labeled Data

Janis Goldzycher, Moritz Preisig, Chantal Amrhein, Gerold SchneiderACL 2023

TL;DR

This paper investigates using Natural Language Inference (NLI) models for hate speech detection in low-resource languages. They find that NLI fine-tuning offers significant improvements over standard methods when limited target language data is available, especially when combined with English hate speech data. The work also explores hypothesis engineering as a way to further boost performance in specific scenarios.

What Problem Does This Paper Address?

Hate speech detection is crucial for online safety, but most research and available datasets are focused on English. This leaves many language communities underserved. Creating labeled datasets for new languages is expensive, time-consuming, and can be psychologically taxing for annotators. Therefore, there's a pressing need for methods that can effectively detect hate speech in languages with scarce labeled data. This paper explores whether Natural Language Inference (NLI) models, which have shown promise in zero-shot and few-shot settings, can be leveraged to address this challenge.

Methods

The authors evaluate several approaches, building upon existing work:

  • Standard Fine-tuning: Training a model directly on the target language hate speech data.
  • Intermediate Fine-tuning on English Hate Speech: Fine-tuning a model on English hate speech data before fine-tuning it on the target language data (as proposed by Röttger et al., 2022a).
  • NLI Fine-tuning: This is the core of their investigation. They explore fine-tuning models on NLI tasks using either:
  • Monolingual NLI data in the target language (M + NLI): Training a monolingual model on target language NLI data.
  • Multilingual NLI data (X + NLI): Training a multilingual model on NLI data from the target language.
  • Full Multilingual NLI data (X + XNLI): Training a multilingual model on the entire XNLI dataset (and MNLI), which encourages cross-lingual transfer.
  • Hypothesis Engineering: This technique, proposed by Goldzycher and Schneider (2022), involves using multiple hypotheses with NLI models to refine predictions. They specifically test three strategies: "Filtering by Target," "Filtering Reclaimed Slurs," and "Filtering Counterspeech."

They systematically compare these approaches across various few-shot settings (0, 20, 200, and 2000 labeled examples) in five target languages.

Datasets

The paper utilizes a combination of established datasets:

  • NLI Datasets:
    • MNLI: A large-scale English NLI dataset.
  • XNLI: A cross-lingual NLI dataset derived from MNLI, with translations in 15 languages.
  • English Hate Speech Datasets (En HS):
    • FEN (Founta et al., 2018): Twitter data.
  • KEN (Kennedy et al., 2020): YouTube, Twitter, and Reddit data.
  • DEN (Vidgen et al., 2021): Adversarial examples.
  • Target Language Hate Speech Datasets (TL HS):
    • BAS19_ES (Basile et al., 2019): Spanish Twitter data.
  • FOR19_PT (Fortuna et al., 2019): Portuguese Twitter data.
  • HAS21_HI (Modha et al., 2022): Hindi Twitter data.
  • OUS19_AR (Ousidhoum et al., 2019): Arabic Twitter data.
  • SAN20_IT (Manuela et al., 2020): Italian Twitter data.
  • Multilingual HateCheck (MHC): A suite of synthetic, evaluation-only datasets for cross-lingual evaluation.

Key Insights and Ideas

  • NLI is a powerful tool for low-resource hate speech detection: Even without any target language hate speech data (zero-shot), NLI-based approaches show significant improvements over standard fine-tuning.
  • Intermediate English fine-tuning is strong, but NLI offers a competitive alternative: While fine-tuning on English hate speech data (Röttger et al., 2022a) remains a strong baseline, NLI fine-tuning (especially with full XNLI) achieves comparable performance, particularly when target language data is scarce.
  • Combining NLI and English fine-tuning yields mixed results: Adding NLI fine-tuning before English hate speech fine-tuning provides minor benefits with very few target language examples but doesn't consistently improve performance as more data becomes available.
  • Hypothesis engineering shines in specific, knowledge-rich scenarios: While not universally beneficial, hypothesis engineering can significantly improve performance when domain knowledge about the target data is available, but no matching English training data exists. This suggests it can be tailored to specific challenges.
  • Domain gap is a challenge for NLI: The authors hypothesize that the domain mismatch between NLI training data and hate speech detection tasks might limit NLI's effectiveness, suggesting future work on bridging this gap.

Key Results

  • Significant improvements with NLI: NLI fine-tuning consistently outperformed standard few-shot fine-tuning in the target language across zero-shot and few-shot settings.
  • English fine-tuning often leads: In most scenarios, fine-tuning on English hate speech data (X + En HS) achieved the highest performance, especially with more target language examples.
  • NLI benefits unseen languages: For Portuguese and Italian (not in XNLI), NLI formulations still showed clear benefits over standard fine-tuning.
  • Hypothesis engineering impact: On the English HateCheck dataset, hypothesis engineering strategies led to clear performance improvements for X + XNLI models. However, on held-out test sets, the strategies often had a negative effect.
  • Reproducibility: The authors successfully reproduced the findings of Röttger et al. (2022a), confirming that intermediate fine-tuning on English hate speech is beneficial.

Limitations and Future Work

Limitations:

  • Limited languages: The study focused on five languages, a small fraction of those that could benefit from such methods.
  • Model dependency: Results are based on a specific multilingual model (XLM-T base), and model size and pre-training domain could influence outcomes.
  • Hypothesis engineering strategies: Only a subset of hypothesis engineering strategies were tested, and potential errors in automatic translation of hypotheses could affect results.

Future Work:

  • Bridging the domain gap: Investigating methods to mitigate the domain gap between NLI training data and hate speech detection tasks, possibly by mixing fine-tuning phases.
  • Model capacity: Evaluating the impact of larger model sizes on hypothesis engineering performance, as their base-sized model might have limited its effectiveness compared to larger models used in prior work.
  • Language-specific hypothesis engineering: Exploring how hypothesis engineering can be better adapted to specific languages and cultures, especially for variations in protected groups.

Summary updated

BibTeX
@inproceedings{goldzycher2023evaluating,
  author = {Janis Goldzycher and Moritz Preisig and Chantal Amrhein and Gerold Schneider},
  title = {{Evaluating the Effectiveness of Natural Language Inference for Hate Speech Detection in Languages with Limited Labeled Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure