TL;DR
This paper tackles the crucial problem of detecting online sexism, particularly implicit forms, by fine-tuning a RoBERTa-based model. The authors achieved competitive results in SemEval-2023 Task 10, demonstrating the effectiveness of transfer learning for this challenging task.
What Problem Does This Paper Address?
Online social platforms, while fostering connection, have unfortunately become breeding grounds for sexism. This hostile environment deters women from participating fully, perpetuating prejudice and discrimination. Detecting sexism, especially subtle or implicit expressions, is vital for creating safer online spaces. SemEval-2023 Task 10, specifically Subtask A, aims to address this by classifying textual content as either sexist or not sexist, with an emphasis on explainability. The challenge lies in moving beyond overt slurs to identify more nuanced forms of gender-based harassment.
Methods
The KingsmanTrio team employed a RoBERTa-Sexism-Classifier, a fine-tuned version of the pre-trained RoBERTa transformer model. The process begins with tokenizing the input text using ROBERTaTokenizer, padding tokens to a uniform sequence length, and then feeding them through RoBERTa's 24 transformer layers. The final layer's output is used by ROBERTaForSequenceClassification to make the binary prediction (sexist/not sexist). This approach leverages the powerful contextual understanding capabilities of RoBERTa, which was trained on a massive and diverse corpus, allowing it to capture complex linguistic relationships.
Datasets
The study utilized the dataset provided for SemEval-2023 Task 10 (EDOS). This dataset comprises 20,000 entries sourced from Gab and Reddit, with each entry being independently annotated by three annotators and reviewed by an expert. For Task A, the dataset was split into training (14,000 entries), validation (2,000 entries), and testing (4,000 entries) sets. The authors note that only 24.3% of the data in their dataset was labeled as sexist, highlighting a potential imbalance.
Key Insights and Ideas
- Transfer learning excels at implicit sexism: The paper strongly suggests that transfer learning models, like RoBERTa, are well-suited for detecting implicit sexism, outperforming models trained from scratch, especially on smaller datasets.
- RoBERTa's advantage: The RoBERTa model demonstrated superior performance compared to BERT, likely due to its training on a larger and more diverse dataset, enabling it to develop more robust language representations.
- Challenges of subtle sexism: The authors acknowledge that their model sometimes misclassifies texts where subtle double standards or dismissive language are used, indicating that comprehending nuanced social dynamics remains a significant challenge.
- Dataset imbalance matters: The relatively low proportion of sexist examples in the training data is identified as a potential reason for some misclassifications.
Key Results
The KingsmanTrio system achieved a Macro F1-Score of 0.8500 on the development set, placing them competitively among the 84 participating systems in SemEval-2023 Task 10. While they narrowly missed the top spot, their performance was only 3.8% behind the leading team. The paper also presents a comparative analysis of various individual models, showing that their fine-tuned BERT+RoBERTa approach outperformed simpler models like Bi-LSTM+GloVe, Dense models, LSTM, CNN, and GRU.
Limitations and Future Work
The authors acknowledge that their system struggled with certain instances of implicit sexism, particularly those involving subtle double standards or dismissive language. They attribute some misclassifications to the insufficient number of sexist posts in the training dataset.
For future work, the authors intend to:
- Incorporate domain-specific features to enhance detection.
- Explore other transformer architectures.
- Expand the detection capabilities to include other modalities beyond text, such as images and videos, as social media content is not solely text-based.
Summary updated