TL;DR
This paper tackles the challenge of detecting online sexism, specifically targeting content directed at women, using transformer models. The authors explore fine-tuning BERT-based models and a semi-supervised approach to leverage large amounts of unlabeled data, ultimately achieving a Macro F1 score of 83.1% on the SemEval-2023 Task 10.
What Problem Does This Paper Address?
Online sexism, particularly when directed at women, is a pervasive issue that contributes to hostile online environments and perpetuates harmful social stigmas. Accurately detecting and understanding this content is crucial for mitigating its impact. SemEval-2023 Task 10 specifically focuses on binary sexism detection (Subtask A), aiming to classify text as either sexist or non-sexist. The challenge lies in the imbalanced nature of available datasets, the subtle and varied forms sexism can take, and the need for models that can generalize well beyond explicitly offensive language.
Methods
The researchers experimented with two main approaches:
- Fine-tuning Pre-trained Language Models: They fine-tuned several transformer models, including BERT, DistilBERT, and RoBERTa, on the provided labeled training data. They also investigated the benefit of using models pre-trained on sentiment analysis tasks, hypothesizing that sexism often carries a negative sentiment.
- Semi-supervised Learning: To overcome the limitations of the relatively small labeled dataset, they employed a semi-supervised approach. This involved using their best-performing fine-tuned model to label a large corpus of unlabeled data from Reddit and Gab. They carefully selected confident predictions from this unlabeled data to augment their training set, aiming to improve model robustness.
- Traditional Machine Learning: As a baseline, they also explored traditional methods like Random Forest and AdaBoost using TF-IDF features, though these yielded less satisfactory results.
Datasets
- SemEval-2023 Task 10 Training Data: 10,000 entries from Gab and 10,000 from Reddit, with 14,000 used for training and 2,000 for development. The training data was imbalanced, with approximately 24% of entries labeled as 'sexist'.
- Additional Unlabeled Data: 1 million entries from Gab and 1 million from Reddit were provided by the task organizers for semi-supervised learning.
Key Insights and Ideas
- Sentiment-tuned models didn't offer a significant boost: Despite the intuition that sexism often involves negative sentiment, fine-tuning BERT models already pre-trained on sentiment analysis tasks did not substantially improve the Macro F1 score for sexism detection. This suggests that sexism detection requires more than just sentiment analysis.
- Semi-supervised learning is key for imbalanced data: Leveraging large amounts of unlabeled data proved crucial. By carefully filtering and adding confidently labeled examples, they significantly expanded their training set and improved performance.
- Thresholding is critical for semi-supervised labeling: The authors found that setting appropriate thresholds for 'sexist' (r=0.95) and 'not sexist' (r=0.015) classes was vital to avoid introducing too much noise while still gaining sufficient training data.
- Traditional ML struggles with nuanced language: TF-IDF features and traditional classifiers like Random Forest performed poorly compared to transformer models, highlighting the need for deep learning to capture the complexities of online language.
Key Results
- The submitted model achieved a Macro F1 score of 83.1% on the SemEval-2023 Task 10 (Subtask A), ranking 55th out of 84 participating teams.
- The best performing model was a BERT base model fine-tuned on a combination of the original labeled data and the semi-supervised augmented data.
- The model achieved an accuracy of 0.871 and a Macro F1 score of 0.831 after semi-supervised training.
- Traditional machine learning classifiers achieved much lower Macro F1 scores, with the best performing Random Forest model reaching 0.71.
Limitations and Future Work
- GPU limitations: The authors were unable to test larger BERT models due to GPU constraints.
- Subtle sexism detection: The model struggled with cases where offensive language was present but not explicitly sexist, and vice-versa. This suggests a need for more nuanced understanding of language.
- Combining subtasks: The paper suggests that combining the binary sexism detection (Subtask A) with the fine-grained classification of sexism types (Subtask B) might lead to better overall performance, especially for detecting specific forms of sexism.
- Further exploration of data augmentation: While semi-supervised learning was effective, exploring other data augmentation techniques could further improve robustness.
Summary updated