TL;DR
This paper tackles the challenge of automatically detecting sexism in online English text using a semi-supervised learning approach. They leverage a pre-trained RoBERTa model, further fine-tune it on domain-specific data, and employ Unsupervised Data Augmentation (UDA) to improve robustness, achieving a competitive Macro-F1 score.
What Problem Does This Paper Address?
The proliferation of social media has unfortunately led to the widespread dissemination of hateful and abusive content, including sexism. Automatically identifying and filtering such content is crucial for creating safer online spaces. SemEval-2023 Task 10, specifically Subtask A, aims to address this by developing systems that can accurately detect whether an English-language social media post is sexist. This is a challenging task due to the nuanced and often subtle nature of sexist expressions, which can manifest in various forms beyond overt insults.
Methods
The core of the DUTIR system is built upon the RoBERTa-base pre-trained language model. Their approach involves three main phases:
- Pre-training: They first further pre-train RoBERTa on a large corpus of unlabeled data from the competition's dataset (Gab and Reddit). This step aims to adapt the model to the specific domain of online social media text, allowing it to learn more domain-specific features and nuances. They denote this domain-adapted model as RoBERTa_A.
- Training (Semi-supervised Learning): This phase combines supervised and unsupervised learning.
- Supervised Learning: The RoBERTa_A model is fine-tuned on the labeled dataset using a standard cross-entropy loss to predict the ground-truth label.
- Unsupervised Data Augmentation (UDA): To leverage the vast amount of unlabeled data and improve model robustness, they employ UDA. This involves taking unlabeled examples, creating augmented versions of them using the Easy Data Augmentation (EDA) technique (which includes synonym replacement, random insertion, random swap, and random deletion), and then training the model to produce consistent predictions for both the original and augmented unlabeled examples. The consistency is enforced by minimizing the Kullback-Leibler (KL) divergence between the prediction distributions of the original and augmented inputs. The final loss function is a combination of the supervised cross-entropy loss and the unsupervised KL divergence loss.
- Prediction: For the final prediction on test data, they utilize a majority voting ensemble method. They train multiple models with different hyperparameter settings and combine their predictions to arrive at the final classification.
Datasets
- Labeled Dataset: Provided by the SemEval-2023 Task 10 organizers, consisting of approximately 20,000 labeled entries from Gab and Reddit.
- Unlabeled Dataset: A large corpus of approximately 2 million unlabeled entries from Gab and Reddit, used for pre-training and semi-supervised learning.
Key Insights and Ideas
- Domain-Specific Pre-training Matters: Further pre-training a general-purpose language model like RoBERTa on domain-specific data (social media text in this case) significantly improves performance. This suggests that adapting the model's understanding to the nuances of the target domain is crucial.
- UDA Enhances Robustness: The Unsupervised Data Augmentation (UDA) approach, by enforcing consistency on augmented unlabeled data, effectively improves the model's generalization capabilities and robustness, especially in semi-supervised settings.
- Majority Voting Offers Marginal Gains: While majority voting with models trained on different hyperparameters provided a slight improvement on the test set, the authors note that the gains were modest. This is attributed to the homogeneity of the base models (only hyperparameters differed, not the architecture).
Key Results
The DUTIR system achieved a Macro-F1 score of 0.8352 on Subtask A of SemEval-2023 Task 10, ranking 41st out of 84 participants.
Key experimental findings include:
- RoBERTa_A (domain-pre-trained) outperformed RoBERTa-base (standard pre-trained) on both development and test sets.
- RoBERTa_A + UDA showed further performance improvements, particularly on the development set, highlighting the effectiveness of their semi-supervised strategy.
- The final RoBERTa_A + UDA + Vote model achieved the best result on the test set, demonstrating the utility of ensemble methods.
Limitations and Future Work
- Majority Voting Homogeneity: The authors acknowledge that the majority voting method yielded only slight improvements because the ensemble consisted of models with similar structures, differing only in hyperparameters. They suggest exploring models with diverse structures for more impactful ensemble gains.
- Future Work: The paper emphasizes the need to explore ways to better utilize the abundant unlabeled data to further enhance model generalization. This remains a key area for future research in this domain.
Summary updated