TL;DR
This paper presents an ensemble approach using transformer models to detect and classify online sexism. By ensembling the best-performing models and carefully tuning hyperparameters, the authors achieved strong results on the SemEval-2023 Task 10, demonstrating the effectiveness of combining multiple models for this challenging NLP task.
What Problem Does This Paper Address?
Online platforms are rife with sexism, which can normalize violence and create hostile environments. Detecting and classifying sexism in text is crucial for mitigating its spread. However, sexism can be subtle, context-dependent, and manifest in various forms, making it a difficult problem for automated systems. SemEval-2023 Task 10 specifically aimed to address this by focusing on explainable detection of online sexism, requiring not just classification but also insights into why a piece of text is deemed sexist. This paper tackles both the binary detection of sexism (Task A) and the categorization of its type (Task B).
Methods
The core of the approach relies on Transformer-based models, a state-of-the-art architecture for Natural Language Processing. For Task A (Binary Sexism Detection), the authors employed an ensemble strategy. This involved:
- Model Selection: A review of various transformer models was conducted, and the top three performing models based on preliminary experiments were selected.
- Hyperparameter Optimization: For these selected models, a meticulous search for optimal hyperparameters was performed using a reduced dataset. This was crucial for maximizing performance.
- Ensembling: The three best-performing models were then combined using a majority vote system. This technique aggregates the predictions of individual models to produce a final, often more robust, prediction.
For Task B (Category of Sexism), the approach was simpler: the single best-performing model from Task A (specifically, deBERTa-v3-base) was chosen and retrained using the same hyperparameters and dataset. The task was framed as a multiclass classification problem.
A key aspect of their methodology was data preprocessing, which included lowercasing text, removing URLs, user mentions, and audio/video links. They also experimented with two emoji treatments: replacing emojis with text and tokenizing emojis to be treated as distinct tokens.
Datasets
The primary dataset used for training, validation, and testing was the EDOS (Explainable Detection of Online Sexism) dataset developed for SemEval-2023 Task 10. This dataset contains English tweets labeled for sexism (Task A) and specific categories of sexism (Task B). The total size is 13,984 tweets, with 10,602 non-sexist and 3,398 sexist.
Key Insights and Ideas
- Ensembling for Robustness: Combining multiple transformer models, even those with similar underlying architectures, can lead to improved performance over individual models. The majority vote ensemble proved effective.
- Hyperparameter Tuning is Critical: The paper strongly emphasizes that careful hyperparameter optimization significantly boosts model performance, especially in complex tasks like sexism detection.
- Model Choice Matters: While ensembling is powerful, selecting strong base models (like
nghuyong/ernie-2.0-base-en,bert-base-uncased, andmicrosoft/deberta-v3-basefor Task A) is foundational. - Data Preprocessing Nuances: Experimenting with how emojis are handled (as text vs. tokens) showed that while it can influence results, the "original" preprocessing (without specific emoji treatment) often yielded competitive or superior performance for the chosen models.
Key Results
- Task A: The ensemble system achieved a macro F1-score of 0.8403 during development and 0.8396 on the test set, ranking 33rd.
- Task B: The
deBERTa-v3-basemodel achieved a macro F1-score of 0.6467 during development and 0.5794 on the test set, ranking 51st. - The ensemble consistently outperformed individual models in performance comparisons.
- The agreement percentage among the three ensemble models was 86%.
Limitations and Future Work
- Task B Performance: The performance on Task B was significantly lower than Task A, which the authors attribute to the smaller training data and the greater number of sexist labels.
- Error Analysis: While some examples of prediction errors were analyzed (e.g., subtle language, sarcasm, or the model misinterpreting intent), a comprehensive understanding of all error causes remains a challenge. The paper notes potential issues like insufficient data, model overfitting, data bias, or incorrect hyperparameter usage.
- Future Directions: The authors suggest that further research could explore more sophisticated ensembling techniques, investigate different data augmentation strategies, and delve deeper into explainability methods to better understand and mitigate model errors.
Summary updated