Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification Using Pre-trained Language Models

Shuohuan Wang, Jiaxiang Liu, Xuan Ouyang, Yu Sun

Abstract

The Galileo team from Baidu achieved top rankings in the SemEval-2020 Task 12 competition by employing a unified multilingual learning approach and a robust knowledge distillation strategy. By fine-tuning XLM-R on a combined dataset of five languages and utilizing an ensemble-based teacher-student model for categorization, the team demonstrated that cross-lingual transfer significantly enhances offensive language detection. Their system ranked first overall in Sub-task A (Identification), as well as first in Sub-tasks B (Categorization) and C (Targeting).

Introduction

As internet anonymity continues to facilitate "cyber-violence," the volume of offensive language on social media has increased dramatically. Manual moderation is slow, expensive, and psychologically taxing for human annotators. To address this, the OffensEval 2020 competition tasked participants with three goals: identifying offensive posts across five languages (Sub-task A), categorizing the type of offense (Sub-task B), and identifying the specific target of the offense (Sub-task C). The primary challenges involve the linguistic nuances of hate speech and the scarcity of labeled data for non-English languages.

Methodology

The Galileo team deployed two distinct strategies to master these tasks:

Multilingual Fine-Tuning

For Sub-task A, rather than building separate monolingual models, the team utilized XLM-R (a cross-lingual version of RoBERTa). They adopted a "unified approach," fine-tuning a single model on a combined dataset of English, Arabic, Danish, Greek, and Turkish. This strategy hypothesized that the model could learn universal patterns of offensiveness that transcend specific vocabularies, allowing higher-resource languages to bolster performance in lower-resource ones.

The multi-lingual pre-training and fine-tuning pipeline for offensive language identification.

Knowledge Distillation with Soft Labels

For Sub-tasks B and C, which were more English-centric, the team employed Knowledge Distillation. They built an ensemble of supervised "teacher" models that produced "soft labels"—probability distributions representing classification confidence. A "student" model (ERNIE 2.0 or ALBERT) was then trained to mimic these probabilities. This approach helps the model capture the inherent ambiguity and "gray areas" often found in subjective offensive speech.

Key Results and Insights

The Galileo system was the only team to rank in the top three across all languages in Sub-task A, achieving an average F1 score of 0.8595 and ranking 1st overall.

Performance comparison of XLM-R models and the final ensemble across five languages.

Key findings from the experimental results include:

  • The Multilingual Advantage: Multilingual fine-tuning improved performance in nearly every language compared to monolingual training. Turkish was the only exception, likely due to its already large training set size.
  • Soft Labels vs. Hard Labels: Training on "soft" probability distributions from an ensemble consistently outperformed training on raw "hard" labels for categorization tasks, suggesting that model uncertainty is a valuable feature for subjective classification.
  • Model Performance: Baidu’s ERNIE 2.0 model consistently outperformed ALBERT in distillation tasks, highlighting the effectiveness of its continual pre-training framework for semantic understanding.

Conclusion

The success of the Galileo team underscores the power of multilingual pre-trained models and knowledge distillation in tackling complex, cross-lingual NLP tasks. While the approach was highly successful, the researchers noted that data imbalances (e.g., 31k Turkish samples vs. 3k Danish samples) still pose challenges. Future work will explore zero-shot performance to determine how well these models generalize to entirely new languages without specific fine-tuning.

Summary updated

BibTeX
@inproceedings{wang2020galileo,
  author = {Shuohuan Wang and Jiaxiang Liu and Xuan Ouyang and Yu Sun},
  title = {{Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification Using Pre-trained Language Models}},
  booktitle = {SemEval @ COLING},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.189},
}
Figure