TL;DR
This paper tackles the challenge of detecting online sexism, a critical issue for online safety, by employing an ensemble of transformer models. They investigate how data augmentation and semi-supervised learning can mitigate class imbalance and improve performance, achieving a strong 10th place in the SemEval-2023 competition for binary sexism detection.
What Problem Does This Paper Address?
The proliferation of social media has unfortunately led to an increase in harmful content, including online sexism. Manually moderating this content is infeasible due to its sheer volume. Therefore, developing accurate and robust automated systems for detecting and classifying sexism is crucial for creating safer online environments. The SemEval-2023 Task 10 (EDOS) specifically addresses this by focusing on identifying sexist content in social media posts. A major hurdle in this task is the significant class imbalance, where non-sexist posts vastly outnumber sexist ones, which can lead to biased models that favor the majority class. This paper aims to address this imbalance and improve detection performance through data augmentation and semi-supervised learning techniques.
Methods
The core of the NLP-LTU team's approach is a voting-based ensemble of three pre-trained transformer-based language models: BERTweet-large, DeBERTa-v3-large, and RoBERTa-large. To combat class imbalance and enhance generalization, they explore two key techniques:
- Data Augmentation: They primarily use back-translation (English to German, back to English, then English to Russian, back to English) to generate synthetic data. This method was chosen for its simplicity and task-agnostic nature. They experiment with two strategies: augmenting only the underrepresented (sexist) class to balance the dataset, or augmenting all classes to effectively double the dataset size.
- Semi-Supervised Learning: Leveraging large amounts of unlabelled, in-domain data provided by the task organizers, they fine-tune a BERTweet-large model on the gold-labelled data. This model is then used to generate "weak labels" for the unlabelled data. Samples with a predicted minority class probability of at least 0.9 are selected to augment the training set.
For classification, they employ a pre-processing step to clean noisy data from sources like Reddit and Gab, followed by the ensemble of fine-tuned models.
Datasets
The primary dataset used is from the SemEval-2023 Task 10 (EDOS) shared task. This dataset includes:
- Labeled data: Used for training and fine-tuning models. The paper highlights its imbalanced nature, particularly for Task A (binary sexism detection), which contains 3398 sexist posts and 10602 non-sexist posts.
- Unlabelled data: Two large datasets, each with one million entries, were provided by the task organizers and utilized for semi-supervised learning.
Key Insights and Ideas
- Domain-Specific Models Shine: Using pre-trained language models (PLMs) that are already fine-tuned on domain-specific data (like BERTweet-large) generally leads to better performance than those pre-trained on more general sources.
- Augmenting All Classes is Often Better: In many cases, augmenting all classes (doubling the dataset) yielded higher classification scores than solely augmenting the minority (sexist) class to balance the dataset. This suggests that increasing the overall data volume can be more beneficial than just addressing the imbalance.
- Semi-Supervised Learning's Potential: With a substantial amount of unlabelled, in-domain data, semi-supervised learning can indeed enhance the performance of certain models, particularly smaller ones.
- Ensembling for Robustness: The use of an ensemble of models, specifically a hard voting ensemble, proved to be effective and contributed to their strong performance.
Key Results
- Task A (Binary Sexism Detection): The NLP-LTU team achieved an F1-macro score of 86.13 on the test set for Task A, ranking them 10th in the competition. Their hard ensemble approach using BERTweet-large, DeBERTa-v3-large, and RoBERTa-large without data augmentation performed best.
- Task B (Sexism Classification): Their ensemble model achieved an F1-macro score of 65.50, ranking 18th.
- Task C (Fine-grained Sexism Vector): For this task, they opted for a single RoBERTa-large model, achieving an F1-macro score of 46.00, ranking 23rd.
- Data Augmentation Impact: While data augmentation was explored, the paper notes that for Task A, the back-translation method did not significantly improve overall performance compared to using the original dataset. They hypothesize this might be due to the generated samples not being diverse enough.
Limitations and Future Work
- Back-Translation Diversity: The authors acknowledge that the back-translation method might not generate sufficiently diverse synthetic data, which could explain its limited impact on improving performance.
- Future Exploration of Augmentation: They plan to explore alternative data augmentation techniques, such as generative models like GPT-2, to create more diverse synthetic sentences and further investigate their impact on balancing and doubling datasets.
- Understanding Augmentation Effects: Further research is needed to understand why augmenting all classes is sometimes more effective than just augmenting the underrepresented class.
- Task C Performance: The performance on Task C was lower, and they suggest that individual fine-tuned models might be more suitable than ensembles for this specific task due to time constraints and potentially less comparable individual model performance.
Summary updated