TL;DR
This paper tackles the challenge of automatically identifying offensive language on social media. The authors propose a BERT-based model enhanced with a novel "maximum self-ensemble" strategy, achieving a strong macro F1 score of 0.913 on the SemEval-2020 Task 12 (OffensEval) English subtask. This approach demonstrates the power of ensemble methods in boosting the performance of pre-trained language models for nuanced NLP tasks.
What Problem Does This Paper Address?
The proliferation of social media has led to an explosion of user-generated content, much of which can be offensive or hateful. Manually moderating this content is a monumental, time-consuming, and emotionally taxing task. Consequently, there's a critical need for automated systems that can accurately identify and classify offensive language. SemEval-2020 Task 12 (OffensEval) aims to address this by providing a benchmark for offensive language identification across multiple languages. This paper focuses specifically on Subtask A, which targets the identification of offensive language in English text.
Methods
The core of the proposed system is the BERT (Bidirectional Encoder Representations from Transformers) model, a powerful pre-trained language representation model. The authors leverage BERT's ability to understand context by fine-tuning it on the offensive language detection task.
To further improve performance, they introduce a maximum self-ensemble strategy. This involves training multiple BERT models with different random seeds. For each input text, each individual BERT model produces a prediction and a confidence score. The ensemble then selects the prediction from the model that exhibits the highest confidence for that specific input. This "max-voting" approach aims to mitigate the sensitivity of BERT fine-tuning to random seeds and training data order, thereby reducing overfitting and enhancing generalization.
Prior to model training, extensive data preprocessing is performed. This includes:
- Replacing "@user" mentions with a generic "username" to remove potentially uninformative tokens.
- Standardizing repeated punctuation (e.g., "!!!") into a single instance with a "repeat" tag.
- Expanding contractions (e.g., "there're" to "there are") to improve semantic understanding.
- Replacing emojis and emoticons with their emotional word equivalents.
- Lemmatizing words to their base form using WordNetLemmatizer to handle variations.
- Converting all tokens to lowercase.
- Removing the "#" symbol from hashtags while retaining the hashtag word itself.
Datasets
The primary dataset used is from SemEval-2020 Task 12 (OffensEval). The organizers provided training and test sets containing a substantial number of sentences (9,073,396 for training and 3,887 for testing). A key characteristic of this dataset is that the labels are not direct binary classifications but rather derived from the average confidence (AVG) and standard deviation (STD) of predictions from several supervised models. The authors define offensive language as text with a calculated confidence score greater than 0.50, using two methods for confidence calculation: $S_1 = AVG + STD$ and $S_2 = AVG$.
Key Insights and Ideas
- Ensembling is Crucial for BERT Fine-tuning: The paper strongly suggests that while BERT is a powerful baseline, its performance can be significantly boosted by ensemble methods, especially when dealing with the inherent variability in fine-tuning.
- Maximum Self-Ensemble Outperforms Single Models: The "max-voting" ensemble strategy, where the prediction from the most confident model is chosen, proved more effective than individual BERT models, even those fine-tuned with different data processing methods.
- Data Preprocessing Matters: The detailed preprocessing steps, particularly handling social media specific elements like emojis, repeated punctuation, and contractions, are vital for preparing the text for BERT and improving its understanding of offensive language.
- Confidence Scores as Labels: The novel approach of deriving labels from AVG and STD confidence scores from pre-existing models highlights a clever way to leverage existing resources and create a more robust training signal.
Key Results
The proposed BERT model with the maximum self-ensemble strategy achieved a macro F1 score of 0.913 on the English subtask A of SemEval-2020 Task 12. This placed them 13th out of 82 participating systems. The results table shows that the ensemble model (MEB) consistently outperformed individual BERT models, regardless of whether the $S_1$ or $S_2$ confidence calculation method was used for data preprocessing. For instance, the MEB model with $S_1+S_2$ preprocessing achieved 0.913, while the best single BERT model achieved 0.908.
Limitations and Future Work
The authors acknowledge that their results, while strong, were not as satisfactory as the top-performing teams on the leaderboard. They attribute this partly to time constraints, which prevented them from submitting all their results.
Future work directions include:
- Further model adjustment and optimization.
- Exploring more ensemble methods.
- Addressing the issue of data imbalance, which is a common challenge in classification tasks.
- Investigating the impact of different data preprocessing techniques on model performance.
- Improving hardware configurations for more extensive experimentation.
- Incorporating more external data to enhance model robustness.
Summary updated