TL;DR
This paper tackles the challenge of detecting and categorizing sexism in social media by employing a novel multi-task learning approach. By framing the task as pairwise text classification with label descriptions and dataset identifiers, the authors achieve strong performance, particularly in detecting sexism, and significant improvements in categorization compared to baseline models.
What Problem Does This Paper Address?
The proliferation of social media has unfortunately led to an increase in harmful language, including sexism. Automatically detecting and understanding this sexism is crucial for content moderation and fostering healthier online environments. However, existing methods often rely on complex transformer models whose decision-making processes can be opaque, hindering explainability. Furthermore, datasets for sexism detection are often scarce and not publicly available, making it difficult to train robust models. The SemEval-2023 Task 10 (EDOS) aims to address this by focusing on sexism detection (Subtask A) and its categorization into coarse (Subtask B) and fine-grained (Subtask C) levels, with the goal of improving model explainability.
Methods
The core of the CL-UZH system is a multi-task learning (MTL) framework built upon the DeBERTa-V3-large model. Instead of traditional multi-task setups that might involve multiple model heads, this approach reformulates each task as a binary pairwise text classification problem. The model receives a concatenated input of a dataset identifier and a label description, followed by the text to be classified. It then predicts whether the given label applies to the text.
The training process is structured in three phases:
- General Abusive Language Model: An initial model is trained on a broad range of abusive language detection datasets.
- Multi-Task EDOS Fine-tuning: The model is then fine-tuned on all three EDOS subtasks (A, B, and C) simultaneously, leveraging the label descriptions and dataset identifiers.
- Target Subtask Fine-tuning: Finally, the model is fine-tuned exclusively on the specific target subtask (A, B, or C) for which it will be used.
Key innovations include:
- Label Descriptions: Using natural language descriptions of labels to guide the model's understanding, enhancing its ability to generalize.
- Dataset Identifiers: Incorporating short abbreviations of datasets to help the model disambiguate between similar labels across different data sources.
- Incremental Fine-tuning: A staged training approach that builds from general language understanding to specific sexism detection and categorization.
Datasets
The primary dataset used is the EDOS 2023 dataset, which contains 20,000 posts from Gab and Reddit, labeled for sexism (binary), coarse-grained categories (4), and fine-grained subcategories (11). The authors also utilize several auxiliary datasets for multi-task learning, including:
- DGHSD (Dynamically Generated Hate Speech Dataset): Contains adversarial examples for hate speech detection.
- MHS (Measuring Hate Speech): Labeled for ten attributes related to hate speech.
- SBF (Social Bias Frames): A collection of Twitter and Reddit data.
- TweetEval: A benchmark for various tweet classification tasks, including emotion, irony, hate speech, offensive language, sentiment, and stance detection.
The authors explicitly state they do not use the unlabelled data provided by the EDOS organizers.
Key Insights and Ideas
- Label Descriptions are Powerful: The ablation study strongly suggests that using label descriptions significantly boosts performance, especially for the more challenging subtasks B and C. This highlights the model's ability to leverage semantic information from the descriptions.
- Multi-Task Learning is Key for Categorization: While label descriptions help across the board, the dramatic improvements in subtasks B and C are attributed to the multi-task learning aspect, indicating that learning related tasks simultaneously is crucial for fine-grained categorization.
- NLI Pre-training Provides a Solid Foundation: Starting with a DeBERTa model already fine-tuned on Natural Language Inference (NLI) tasks provides a performance boost across all subtasks, suggesting that the model's ability to understand sentence relationships is beneficial.
- Dataset Identifiers' Impact is Unclear: The authors found mixed results for dataset identifiers, with some subtasks showing slight improvements and others slight decreases. This suggests their utility might be task- or dataset-dependent and warrants further investigation.
- Class Imbalance is a Major Hurdle: The baseline single-task models perform well on Subtask A but poorly on B and C. This is attributed to the severe class imbalance in the latter subtasks, where scarce classes can drastically reduce the macro-F1 score. Upsampling low-frequency classes did show positive effects.
Key Results
The CL-UZH system achieved competitive results in the official evaluation:
- Subtask A (Sexism Detection): F₁-score of 85.9% (Rank 13/84)
- Subtask B (Coarse-grained Categorization): F₁-score of 64.8% (Rank 19/69)
- Subtask C (Fine-grained Categorization): F₁-score of 44.9% (Rank 26/63)
The ablation study demonstrated substantial gains from multi-task learning with label descriptions, particularly for subtasks B and C, where performance more than doubled compared to single-task baselines. Adding a second fine-tuning phase for the target subtask also improved performance on B and C but slightly decreased it on A.
Limitations and Future Work
- Performance Drop with More Classes: The authors note that their method's performance decreases as the number of classes increases, especially for subtasks B and C. This suggests that while multi-task learning helps, it doesn't fully overcome the challenges of fine-grained categorization with many classes.
- Dataset Identifier Uncertainty: The impact of dataset identifiers remains unclear, suggesting future work could explore more sophisticated ways to signal dataset provenance or investigate why they don't consistently help.
- Class Balancing: While upsampling helped, the inherent class imbalance in subtasks B and C remains a significant challenge. Future work could explore more advanced techniques for handling imbalanced data in multi-class settings.
- Explainability: While the task aims for explainability, the paper focuses on detection and categorization performance. Further research could delve into how the learned label descriptions and multi-task structure contribute to model interpretability.
Summary updated