Abstract
Nowadays, offensive content in social media has become a serious problem, and automatically detecting offensive language is an essential task. In this paper, we build an offensive language detection system, which combines multi-task learning with BERT-based models. Using a pre-trained language model such as BERT, we can effectively learn the representations for noisy text in social media. Besides, to boost the performance of offensive language detection, we leverage the supervision signals from other related tasks. In the OffensEval-2020 competition, our model achieves 91.51% F1 score in English Sub-task A, which is comparable to the first place (92.23%F1). An empirical analysis is provided to explain the effectiveness of our approaches.
Introduction
Social media platforms are increasingly plagued by offensive content, ranging from personal insults to coordinated hate speech. Automatically identifying such content is a "wicked problem" for Natural Language Processing (NLP) due to the inherent noise in social media text—characterized by typos, slang, hashtags, and emojis.
The "Kungfupanda" team from HKUST addressed this challenge within the context of the OffensEval-2020 competition (SemEval-2020 Task 12). The competition defines a three-level hierarchy for offensive language detection:
- Sub-task A: Offensive Language Detection (Offensive vs. Not Offensive).
- Sub-task B: Categorization of Offense (Targeted Insult vs. Untargeted Profanity).
- Sub-task C: Target Identification (Individual, Group, or Other).
The primary goal is to improve the accuracy of Sub-task A by leveraging the logical relationships between these hierarchical levels.
Methodology
The proposed system utilizes BERT (Bidirectional Encoder Representations from Transformers) as a backbone to capture deep contextual representations of text.
Data Pre-processing
To mitigate the noise in Twitter data, the team implemented several pre-processing steps:
- Emoji Conversion: Emojis were converted into descriptive text (e.g., ✌️ to "victory hand").
- Hashtag Segmentation: Hashtags were split into individual words (e.g.,
#KeithEllisonAbuseto "keith ellison abuse"). - User Mention Normalization: Multiple user mentions were replaced with a single generic token to reduce redundancy.
Multi-Task Learning (MTL) Framework
Instead of training a separate model for each sub-task, the researchers developed a Multi-Task Learning (MTL) framework. As shown in Figure 2, the model shares a common BERT backbone across all tasks. Each task has its own specific "head" consisting of a Long Short-Term Memory (LSTM) layer followed by a Feed-Forward Network (FFN) and a Softmax layer.
During training, the model minimizes a weighted sum of losses from all three tasks. This approach allows the model to "eavesdrop" on related signals; for instance, knowing a post is "Targeted" (Sub-task B) strongly implies it is "Offensive" (Sub-task A).
Datasets
The study employed two datasets:
- OLID (Offensive Language Identification Dataset): 14,100 manually annotated tweets, serving as high-quality ground truth.
- SOLID (Semi-Supervised Offensive Language Identification Dataset): A massive dataset of over 9 million tweets with labels generated via co-training. Despite its size, the authors found that the noise in SOLID's semi-supervised labels limited its utility for pre-training.
Results and Analysis
The Kungfupanda system performed exceptionally well in the OffensEval-2020 competition, achieving a 91.51% macro-F1 score on English Sub-task A and placing 7th out of 85 submissions.
Ablation studies demonstrate that the MTL approach significantly outperforms the standard single-task BERT baseline. Specifically, the BERT + MTL (Ensemble) model reached the highest performance (0.8382 F1 on OLID), confirming that joint learning across hierarchical tasks enhances the model's ability to generalize.
Conclusion
The integration of BERT with a Multi-Task Learning framework proved highly effective for offensive language detection. By capturing contextual nuances and leveraging hierarchical task dependencies, the model achieved competitive results. Future work will explore more complex task interactions and asynchronous optimization techniques to better handle varying task difficulties and noisy gradients.
Summary updated