Abstract
The proliferation of offensive content on social media necessitates robust automated detection systems. In this paper, the LT@Helsinki team evaluates the performance of multilingual versus language-specific Transformer-based models for SemEval-2020 Task 12 (OffensEval 2020). We find that while multilingual BERT (mBERT) provides a strong baseline, language-specific models like Nordic BERT offer superior performance by capturing linguistic nuances. Our approach secured first place in Danish offensive language identification and second place in English offense target identification.
Introduction
As social media platforms host billions of users, human moderation of offensive content has become unscalable. The OffensEval 2020 challenge addressed this by tasking participants with identifying offensive language across five languages (Arabic, Danish, English, Greek, and Turkish) and categorizing the targets of such offense (individuals, groups, or others). A central question in our research was whether a single multilingual model could suffice for these diverse tasks or if language-specific pre-training is necessary for optimal performance.
Methods
We compared traditional machine learning baselines against modern Transformer architectures. Our baseline consisted of a Random Forest classifier utilizing TF-IDF and surface-level features, such as URL counts, mentions, and abusive term lexicons.
For our primary approach, we utilized several BERT variants:
- Nordic BERT: Pre-trained on Danish Wikipedia and forum data, used for Danish tasks.
- Multilingual BERT (mBERT): Used for Arabic and Turkish.
- English BERT-Base: Used for English target identification (Sub-task C).
- Translation-based BERT: For Greek, we translated the dataset into English and applied an English BERT model.
To address class imbalance, particularly in target identification where "Individual" (IND) targets predominate, we employed over-sampling techniques to balance the training sets.
The Power of Specialization
A key finding of our study was the significant performance gap between multilingual and language-specific models. In experiments on the Danish development set, Nordic BERT outperformed mBERT by a substantial margin. Interestingly, even a well-tuned Random Forest baseline outperformed the generic multilingual BERT model, highlighting the importance of language-specific context for lower-resource languages.
We also observed that minimal pre-processing yielded the best results for BERT-based models. Retaining elements like @USER mentions and emojis proved critical, as they often provide the necessary context to identify the target of an insult.
Key Results
Our team achieved top-tier rankings across several sub-tasks. Most notably, we ranked 1st in Danish offensive language identification (Sub-task A) and 2nd in English offense target identification (Sub-task C).
While our models performed well on majority classes, identifying the "Other" (OTH) target category remained challenging due to its sparsity in the training data, often leading to misclassification as "Individual" (IND) due to the sheer volume of IND examples.
Conclusion
The LT@Helsinki participation in OffensEval 2020 underscores the superiority of language-specific models over general multilingual ones for offensive language detection. Future work will explore language-specific models for Arabic and Turkish, as well as ensemble methods to further improve robustness against class imbalance.
Summary updated