TL;DR
This paper presents UMUTeam's approach to the SemEval-2023 Task 10, focusing on fine-grained and explainable detection of sexism in online text. They achieved strong results by extending a pre-trained RoBERTa-large model with unlabeled sexism-related data before fine-tuning it for specific classification tasks, demonstrating the effectiveness of domain adaptation.
What Problem Does This Paper Address?
The internet has become a breeding ground for sexism, negatively impacting individuals and making online spaces less accessible. While automated tools exist to detect sexist content, they often lack precision and fail to explain why a piece of content is considered sexist. This limits their interpretability and usefulness. The SemEval-2023 Task 10, "Explainable Detection of Online Sexism (EDOS)," aimed to address this by challenging participants to develop classifiers that not only identify sexism but also provide detailed explanations for their classifications on platforms like Gab and Reddit. The task was structured into three hierarchical subtasks: binary sexism detection, categorization of sexism, and fine-grained vector classification of sexist content.
Methods
UMUTeam's approach centers on leveraging and adapting pre-trained Transformer models. They utilized RoBERTa-large, a powerful masked language model (MLM), as their base. The core of their strategy involved two main phases:
- Domain Adaptation (MLM Pre-training): They augmented the existing pre-trained RoBERTa-large model by further pre-training it on a large, unlabeled dataset of sexism-related text from Gab and Reddit. This step aims to imbue the model with a deeper understanding of the nuances and language patterns specific to online sexism, improving its generalization capabilities.
- Task-Specific Fine-Tuning: After the domain adaptation, the modified RoBERTa-large model was fine-tuned on the labeled datasets provided for each of the three EDOS subtasks (Task A, B, and C). This process tailors the model to perform the specific classification objectives of each task.
Preprocessing involved standard text cleaning, including replacing hashtags and mentions, removing emojis, and expanding contractions. For tokenization, they used RoBERTa-large's native byte-level BPE. Hyperparameter optimization was performed using RayTune with a Tree of Parzen Estimators approach to find the best configuration for weight decay, batch size, training epochs, and learning rate.
Datasets
The primary datasets used were provided by the EDOS shared task organizers:
- Labeled Datasets: These were used for fine-tuning the models for each task. The dataset consisted of 20,000 entries, split between Gab and Reddit, with labels adjudicated by trained annotators and experts.
- Task A: Binary classification (sexist/not sexist).
- Task B: Four-class classification for sexist posts (threats, derogation, animosity, prejudiced discussions).
- Task C: Eleven-class classification for fine-grained sexist content vectors.
- Unlabeled Datasets: Two large unlabeled auxiliary datasets were provided: 1 million Gab posts and 1 million Reddit posts. These were crucial for the domain adaptation phase.
Key Insights and Ideas
- Domain Adaptation is Key: The paper strongly suggests that adapting large pre-trained language models to the specific domain of online sexism (using unlabeled data) significantly improves performance compared to using a generic pre-trained model. This is evident in their results, where the domain-adapted model consistently outperformed the base RoBERTa-large.
- Imbalance is a Challenge: The dataset for Task A (binary sexism detection) was found to be imbalanced, with significantly more non-sexist posts than sexist ones. This highlights a common challenge in real-world NLP tasks and underscores the importance of evaluation metrics like Macro F1, which give equal weight to all classes.
- Hierarchical Structure Matters: The task's hierarchical structure (Task A -> Task B -> Task C) means that errors in earlier stages can propagate. The authors observed that errors in classifying prejudiced discussions and animosity were common in Task B, which likely impacted the performance on Task C, as Task C further refines these categories.
- Semantic Nuances are Difficult: The paper notes that semantic and syntactic similarities between categories (e.g., animosity and derogation) make fine-grained classification challenging, even with domain adaptation.
Key Results
UMUTeam achieved competitive results across all three tasks:
- Task A (Binary Sexism Detection): Ranked 24th out of 84 teams with a Macro F1 score of 0.8495. This was only 2.51% worse than the top-performing team.
- Task B (Category of Sexism): Ranked 23rd out of 70 teams with a Macro F1 score of 0.6395.
- Task C (Fine-grained Vector of Sexism): Ranked 13th out of 63 teams with a Macro F1 score of 0.4793.
The domain-adapted RoBERTa-large model showed improvements over the generic model: +0.691% in Task A, +0.817% in Task B, and +4.44% in Task C. Confusion matrices revealed that the model often confused prejudiced discussions and animosity in Task B, and subcategories within these in Task C.
Limitations and Future Work
- Emoji Exclusion: The authors acknowledge that their system did not utilize emoji features. They plan to incorporate these, along with other figurative language features, in future work, believing they can improve the system's understanding of nuanced language.
- Data Imbalance: While they addressed it with Macro F1, the inherent data imbalance in Task A remains a challenge.
- Generalization to Other Platforms: The current work focuses on Gab and Reddit. Future work could explore generalization to other social media platforms with different linguistic characteristics.
- Explainability: While the task focused on "explainable detection," the paper primarily reports classification performance. Further research could delve deeper into the mechanisms of explainability provided by their adapted model.
Summary updated