Multilingual Offensive Language Identification with Cross-lingual Embeddings

Tharindu Ranasinghe, Marcos ZampieriEMNLP 2020 · DOI

Abstract

Automated moderation is essential for managing the flood of offensive content on social media, yet the field suffers from a significant "English bias." This paper demonstrates how to effectively identify offensive language in low-resource languages—specifically Bengali, Hindi, and Spanish—by leveraging large English datasets through transfer learning. Utilizing XLM-R, a state-of-the-art cross-lingual transformer model, we achieve results that outperform previous specialized systems, even when transferring knowledge across different social media platforms and classification tasks.

Introduction

The proliferation of offensive content on social media necessitates automated detection systems. However, most high-quality, annotated datasets are in English, leaving languages with fewer resources struggling to build reliable systems. The challenge extends beyond translation; it involves capturing cultural nuances and platform-specific slang. Furthermore, datasets often use disparate labeling schemes (e.g., "hate speech" vs. "aggression"), complicating direct knowledge transfer.

Methodology

We propose a transfer learning pipeline based on XLM-R (Cross-lingual Language Model - RoBERTa), which is pre-trained on a massive corpus of 104 languages. This makes it uniquely suited for cross-lingual tasks where source and target languages differ significantly.

Conceptual diagram of the transfer learning process from a high-resource English model to a low-resource language model.

The methodology employs two primary strategies:

  1. Inter-language Transfer Learning: The model is first trained on the English OLID dataset. For languages with similar binary labels (Hindi and Spanish), the entire model weights, including the classification layer, are used to initialize training for the target language.
  2. Inter-task and Inter-language Transfer Learning: For Bengali, which utilizes a three-way classification (Overtly Aggressive, Covertly Aggressive, Non-Aggressive) instead of binary labels, we transfer only the transformer weights and initialize a new classification head to handle the specific task requirements.

Datasets and Experimental Setup

The study utilizes four primary benchmarks to test the robustness of the transfer learning approach:

  • English (OLID): 14,100 tweets used as the source for transfer learning.
  • Hindi (HASOC 2019): 8,000 tweets labeled for hate/offensive content.
  • Spanish (HatEval): 6,600 tweets focused on hate speech against immigrants and women.
  • Bengali (TRAC-2): 4,000 Facebook comments. This dataset is particularly challenging as it is both off-domain (Facebook vs. Twitter) and off-task (three classes vs. two) compared to the English source data.

Key Results

The XLM-R transfer learning (TL) approach established new benchmarks across all three target languages. In every instance, the "Transfer Learning" version of the model outperformed versions trained only on the target language, proving that English data provides a valuable performance boost even for distant languages.

Comparison of Macro F1 (M F1) and Weighted F1 (W F1) scores across Bengali, Hindi, and Spanish for various models, highlighting the superior performance of XLM-R (TL).

Key Insights

  • XLM-R vs. mBERT: XLM-R significantly outperforms Multilingual BERT (mBERT), suggesting its larger scale and training objectives provide more robust cross-lingual representations for nuanced social tasks.
  • Domain Robustness: The success on the Bengali dataset indicates that the underlying representation of "offensiveness" transcends specific platform slang (e.g., Twitter vs. Facebook).
  • Label Mapping: English "offensive" labels are general enough to serve as a "universal" starting point for more specific sub-tasks like "aggression" or "hate speech" in other languages.

Limitations and Future Work

While the results are impressive, error analysis via confusion matrices shows that the model still struggles to distinguish between "Overtly Aggressive" and "Covertly Aggressive" content in Bengali. This suggests that sarcasm and subtle aggression remain difficult for current cross-lingual embeddings to capture fully. Future work will involve scaling the approach with the massive SOLID dataset (9 million tweets) and expanding support to languages such as Arabic, Danish, and Turkish.

Summary updated

BibTeX
@inproceedings{ranasinghe2020multilingual,
  author = {Tharindu Ranasinghe and Marcos Zampieri},
  title = {{Multilingual Offensive Language Identification with Cross-lingual Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.470},
}
Figure