TL;DR
This paper tackles the critical problem of detecting and classifying online sexism, a pervasive issue on social media. The authors propose a fine-tuned RoBERTa model that achieves state-of-the-art performance across three sub-tasks: binary sexism detection, sexism categorization, and fine-grained sexism classification, demonstrating its effectiveness in identifying nuanced forms of online misogyny.
What Problem Does This Paper Address?
Social media platforms, while connecting people globally, have unfortunately become breeding grounds for sexist content, particularly targeting women. This online sexism can manifest in various subtle and overt forms, ranging from direct insults and threats to more insidious stereotyping and objectification. Identifying and understanding these different facets of sexism is crucial due to its potential to cause significant harm, including contributing to mental health issues like eating disorders and depression. Existing research has often focused on explicit hate speech or specific types of sexism, leaving a gap in comprehensive, nuanced detection and classification systems. This paper addresses this by proposing a system for the SemEval-2023 Task 10, aiming to detect and classify online sexism in a fine-grained manner.
Methods
The core of the proposed solution is a fine-tuned RoBERTa (Robustly optimized BERT approach) model. RoBERTa, a powerful pre-trained language model, was chosen for its ability to capture complex linguistic nuances. The authors employed a multi-task learning approach, where a single RoBERTa model is used to address all three sub-tasks simultaneously. This involves encoding the initial representation of social media posts using RoBERTa and then feeding these embeddings into three distinct Multilayer Perceptrons (MLPs), each tailored to one of the sub-tasks. This shared architecture allows for efficient learning and parameter sharing, reducing the overall model size. The authors experimented with several baseline models, including SBERT, BERT, and BERT-TFIDF, to benchmark their fine-tuned RoBERTa approach.
Datasets
The study utilizes a dataset of 20,000 social media posts, sourced from Reddit and Gab. This dataset was meticulously labeled by three trained annotators, with a final decision made by one of two experts to ensure high quality. The dataset is structured to support three distinct tasks:
- Task A (Binary Sexism Detection): Classifies posts as either "sexist" or "non-sexist."
- Task B (Category of Sexism): Divides sexist content into four categories: Threats, Derogation, Animosity, and Prejudiced discussion.
- Task C (Fine-grained Vector of Sexism): Further breaks down sexist content into 11 specific sub-categories, providing a more granular understanding of the nature of the sexism.
The dataset splits for training, validation, and testing are provided for each task.
Key Insights and Ideas
- RoBERTa's Superiority for Nuanced Sexism: The fine-tuned RoBERTa model consistently outperformed other models, including BERT and SBERT, across all three tasks, highlighting its effectiveness in capturing the subtle and varied expressions of sexism.
- Multi-task Learning Efficiency: Employing a single multi-task learning model for all three sub-tasks proved to be an effective strategy, sharing learned representations and reducing computational overhead without sacrificing performance.
- TF-IDF's Limited Impact on Multiclass Tasks: While BERT-TFIDF showed some improvement for binary classification (Task A), its inclusion did not benefit, and in some cases, even hindered performance on the more complex multi-class tasks (Task B and C). This suggests that for fine-grained classification, the contextual embeddings from RoBERTa alone are more crucial than traditional TF-IDF features.
- The Challenge of Fine-Grained Classification: Task C, with its 11 fine-grained categories, presented the most significant challenge, with models achieving lower F1 scores compared to Tasks A and B. This underscores the difficulty in precisely categorizing the diverse and often overlapping forms of online sexism.
Key Results
The fine-tuned RoBERTa model achieved the highest Macro-F1 scores across all tasks:
- Task A (Binary Sexism Detection): 83.64% on the development set and 79.43% on the test set.
- Task B (Category of Sexism): 65.88% on the development set and 61.91% on the test set.
- Task C (Fine-grained Sexism Classification): 33.20% on the development set and 29.9% on the test set.
These results demonstrate a significant improvement over baseline models, particularly for binary sexism detection. The confusion matrices provided offer a detailed look at the model's performance on specific categories, revealing areas of strength and weakness.
Limitations and Future Work
The authors acknowledge that the performance on Task C, the fine-grained classification, is still relatively low, indicating the inherent difficulty of this task. They suggest that exploring how this model behaves on different types of datasets would be a valuable future direction. Further research could also focus on improving the explainability of the model's predictions, which is a key aspect of the SemEval task, and potentially developing more robust methods for handling the nuances of implicit and subtle sexism.
Summary updated