TL;DR
This paper tackles the critical challenge of automatically detecting and explaining online sexism, a pervasive issue on social media. The authors leverage a suite of powerful NLP transformer models, including BERT, ALBERT, RoBERTa, DistilBERT, and XLNet, to achieve this. Their approach, particularly an ensemble of these models, demonstrates promising results in identifying various forms of sexism, contributing to safer online environments.
What Problem Does This Paper Address?
The proliferation of online platforms and social media has unfortunately amplified the problem of sexism, leading to widespread online harassment and hate speech, disproportionately targeting women. Identifying and understanding these sexist expressions is crucial for creating safer online spaces. However, online sexism can manifest in subtle and varied ways, making its detection a complex task. This paper addresses the need for robust and explainable methods to automatically identify and classify different forms of online sexism, as presented in SemEval-2023 Task 10.
Methods
The core of this work lies in the application and ensemble of various state-of-the-art Natural Language Processing (NLP) transformer models. The authors experimented with:
- ALBERT: A "lite" version of BERT, designed to be more parameter-efficient while maintaining strong performance.
- BERT (Bidirectional Encoder Representations from Transformers): A foundational model for many NLP tasks, known for its ability to understand context.
- RoBERTa (Robustly Optimized BERT Approach): An optimized version of BERT, trained on more data with larger batch sizes and longer training times.
- DistilBERT: A distilled, smaller, faster, and lighter version of BERT, achieved through knowledge transfer.
- XLNet: A generalized autoregressive pretraining method that combines the strengths of autoregressive and autoencoding models.
The authors pre-processed the noisy social media data by removing stopwords, URLs, mentions, HTML tags, emoticons, and special characters, while also expanding contractions and lemmatizing text. For the detection tasks, they initially trained individual transformer models. Crucially, they found that combining the predictions of multiple models through ensemble methods (specifically, majority voting and a more sophisticated approach involving summing prediction probabilities) significantly improved performance, especially for the more granular classification tasks.
Datasets
The study utilized the dataset provided for SemEval-2023 Task 10. This dataset comprises 20,000 entries in total, split between Gab and Reddit. The data was divided into:
- Training data: 14,000 entries (70%)
- Development data: 2,000 entries (10%)
- Test data: 4,000 entries (20%)
The dataset supports three hierarchical subtasks:
- Task A: Binary classification (sexist vs. not sexist).
- Task B: Four-class classification (threats, derogation, animosity, prejudiced discussions).
- Task C: Eleven-class classification for fine-grained sexist vectors.
Key Insights and Ideas
- Ensembling is Key for Granularity: While individual transformer models show promise, their collective output through ensembling proved to be significantly more effective, especially for the more challenging fine-grained classification tasks (Tasks B and C). This suggests that different models capture different nuances of sexist language.
- Data Pre-processing is Crucial: The noisy nature of social media data necessitates thorough pre-processing. Removing irrelevant elements and standardizing text significantly aids the transformer models in learning meaningful patterns.
- Explainability is a Goal, but Performance is Paramount: The task itself emphasizes explainability, but the authors' primary focus and reported metrics are on achieving high performance (macro F1 scores). While their ensemble approach likely offers some level of interpretability through model agreement, deeper explainability techniques were not the central focus of their submission.
Key Results
The authors reported the following macro F1 scores for their final submissions:
- Task A: 0.7937
- Task B: 0.5284
- Task C: 0.2674
These scores placed their team 66th for Task A, 57th for Task B, and 53rd for Task C on the SemEval 2023 leaderboard. They observed that their ensemble model generally outperformed individual models, particularly for Task B and C. For instance, their ensemble model achieved a macro F1 of 0.580871 for Task B, compared to the best individual model (XLNet) at 0.537929.
Limitations and Future Work
The authors acknowledge that the performance on Task C, the most granular classification, is significantly lower than on Task A. This highlights the inherent difficulty in distinguishing between very fine-grained categories of sexism.
For future work, they propose:
- Exploring other ensembling techniques: Specifically, averaging the probabilities of the models.
- Experimenting with external resources: Incorporating external knowledge bases or datasets could potentially enrich the models' understanding.
- Employing different data augmentation techniques: Generating more training data artificially could help improve robustness and performance, especially for under-represented categories.
Summary updated