TL;DR
This paper tackles the challenge of automatically detecting sexist text on social media by employing an ensemble of 18 transformer-based models, including variations of DeBERTa and BERT. Their approach achieved a strong F1-score of 84.55% on the SemEval-2023 Task 10 test set, highlighting the effectiveness of combining multiple models for this complex NLP task.
What Problem Does This Paper Address?
Sexist language is a pervasive issue on social media, perpetuating harmful stereotypes and marginalizing groups, particularly women. Automatically identifying and addressing this language is crucial for promoting gender equality. The SemEval-2023 Task 10 specifically focuses on this challenge, providing a platform for researchers to develop and evaluate methods for detecting sexist text in online communication. Previous research has explored various approaches, but the subtle and nuanced nature of language use makes this a complex problem.
Methods
The core of this paper's approach is an ensemble modeling strategy. They combined predictions from 18 different pre-trained transformer models. Specifically, they fine-tuned variations of DeBERTa-v3-base and BERT-based models (including one trained on hate speech detection, "hateBERT") on the task's provided labeled and unlabeled data.
The ensemble method used was a soft-ensemble, where the probabilities from each individual model were combined using a weighted average to produce the final prediction. This approach is favored because it takes into account the confidence of each model's prediction, unlike a hard-ensemble which simply picks the most frequent prediction. They also experimented with a hard-ensemble but found the soft-ensemble to be superior.
Datasets
The primary dataset used for training and evaluation was the SemEval-2023 Task 10 Subtask A dataset. This dataset consists of 20,000 entries sampled from Gab and Reddit.
- Training data: 14,000 entries (70% of the total), with 3,398 labeled as sexist.
- Development data: 2,000 entries (10% of the total).
- Test data: 4,000 entries (20% of the total).
Additionally, the organizers provided two unlabeled datasets (1 million entries each from Gab and Reddit) which were used for pre-training some of the models.
Key Insights and Ideas
- Ensembling is key: The paper strongly emphasizes that combining multiple models significantly outperforms individual models. This suggests that different models capture different aspects of sexist language, and their collective intelligence leads to more robust predictions.
- Soft-ensemble superiority: The soft-ensemble method, which considers model confidence, yielded better results than a hard-ensemble. This highlights the importance of not just what a model predicts, but how confident it is in that prediction.
- Max length matters: For fine-tuning transformer models, the
max_lengthparameter (the maximum number of tokens a model processes) was found to be influential. Interestingly, shortermax_lengthvalues sometimes led to better performance, suggesting that a more concise input can provide focused context without introducing noise. - Data augmentation's limited impact: Despite exploring various data augmentation techniques, the authors found that they did not lead to significant improvements. The gains in performance were primarily attributed to the ensemble approach itself.
Key Results
The authors' ensemble approach achieved:
- An F1-score of 84.92% on the development dataset.
- An F1-score of 84.55% on the test dataset. This performance ranked them 30th out of 89 participating teams in the SemEval-2023 Task 10.
Limitations and Future Work
The authors acknowledge that their system struggled with identifying certain types of sexist sentences, particularly those involving:
- Immutable gender differences and gender stereotypes.
- Incitement and encouragement of harm.
- Dehumanizing attacks and overt sexual objectification.
These types of sexism often rely on subtle language, implicit meanings, and a deep understanding of context and social issues, which are challenging for current models.
For future work, they plan to explore:
- Additional feature engineering techniques.
- Alternative model selection methods to further improve the ensemble's performance.
- Determining the optimal number of models to include in an ensemble.
Summary updated