TL;DR
This paper explores sexism detection in online text, a crucial but challenging task due to the nuanced and contextual nature of sexism. The authors investigate various RoBERTa-based models and techniques, ultimately finding that an ensemble of Sup-SimCSE-ROBERTa with dropout and XLNet achieves the best performance, highlighting the power of combining models and data augmentation for this complex problem.
What Problem Does This Paper Address?
Identifying sexism in online content is a significant challenge. Sexism can manifest in subtle ways, often requiring contextual understanding, making automated detection difficult. While existing tools can categorize hate speech, they often lack interpretability and struggle with the nuances of sexism against women. This paper tackles the problem of building more effective and robust sexism detection models, specifically for online text, by exploring various deep learning approaches and data manipulation techniques.
Methods
The authors experimented with several deep learning models, including TextCNN, BERT, RoBERTa, XLNet, and Sup-SimCSE-ROBERTa. They focused on RoBERTa-based architectures, particularly Sup-SimCSE-ROBERTa, which leverages contrastive learning to better distinguish between different labels.
To improve performance, they explored several techniques:
- Pseudo-labeling: Using a model trained on labeled data to predict labels for unlabeled data, then incorporating high-confidence predictions back into the training set.
- Data Augmentation: Employing methods like synonym substitution and random insertion to increase the size and diversity of the sexist training data.
- Dropout: A regularization technique to prevent overfitting and improve generalization. They specifically explored applying dropout to sexist data.
- R-drop: A more advanced dropout method that aims to mitigate inconsistencies between training and testing by calculating KL divergence between dropout-generated samples.
- Model Ensemble: Combining the predictions of multiple models to improve overall performance and stability. Their final ensemble combined Sup-SimCSE-ROBERTa and XLNet.
- Focal Loss: Investigated as an alternative to cross-entropy loss to address the issue of class imbalance in the dataset.
Datasets
The primary dataset used in this task was provided by the SemEval-2023 Task 10 organizers. It consists of 20,000 entries, with 10,000 from Gab and 10,000 from Reddit. The training data comprised 14,000 entries, of which 3,398 were labeled as sexist (approximately 24%). The remaining data was split into validation and test sets. Additionally, two million unlabeled entries were provided by the task organizers.
Key Insights and Ideas
- RoBERTa's Strength: RoBERTa-based models, especially Sup-SimCSE-ROBERTa, generally outperformed simpler models like TextCNN, demonstrating the effectiveness of larger, pre-trained transformer architectures for this task.
- Data Imbalance is a Major Hurdle: The dataset's significant imbalance (only 24% sexist samples) was a key challenge, necessitating techniques like pseudo-labeling and Focal Loss.
- Ensembling is Powerful: Combining the strengths of different models, particularly Sup-SimCSE-ROBERTa and XLNet, proved to be a highly effective strategy for boosting performance.
- Dropout's Nuance: While dropout generally improved performance, the specific application (e.g., on sexist data only) and the use of R-drop had varying effects, suggesting that careful tuning is required.
- Data Augmentation's Mixed Results: While data augmentation was explored, its effectiveness was not as pronounced as expected, with some variations even negatively impacting performance, suggesting that the augmentation strategies might not have been optimally tuned for this specific task.
Key Results
The authors achieved a Macro-F1 score of 0.823 on the final test set for SubTask-A. Their best-performing model was an ensemble of Sup-SimCSE-ROBERTa (with dropout applied to sexist data) and XLNet. Individually, Sup-SimCSE-ROBERTa achieved the highest Macro-F1 score of 0.8390 on the local test set, followed closely by XLNet (0.8224) and RoBERTa (0.8272). The ensemble approach (SRX) achieved a Macro-F1 score of 0.8441 on the local test set.
Limitations and Future Work
The authors acknowledge that some experimental methods, particularly data augmentation, did not yield the expected improvements and may have been negatively impacted by objective errors or suboptimal tuning. They suggest that focusing on specific iconic sexism words and symbols might be more effective than general augmentation.
Future work includes:
- Further improving methods like pseudo-labeling.
- Exploring larger RoBERTa models to achieve even better results.
- Investigating more sophisticated data augmentation techniques.
Summary updated