AdamR at SemEval-2023 Task 10: Solving the Class Imbalance Problem in Sexism Detection with Ensemble Learning

Adam Rydelek, Daryna Dementieva, Georg GrohACL 2023

TL;DR

This paper tackles the challenge of detecting online sexism, a task complicated by imbalanced datasets and the need for explainability. The authors propose an ensemble learning approach using Transformer models, achieving top 40% performance across all three subtasks of the SemEval-2023 Task 10. Their work highlights the effectiveness of combining multiple models and strategies to combat class imbalance in this critical area.

What Problem Does This Paper Address?

Online sexism is a pervasive issue that can discourage participation and pose a threat to individuals and groups. Automating its detection is crucial, but existing systems often lack transparency, failing to explain why a piece of text is flagged as sexist. The SemEval-2023 Task 10, "Explainable Detection of Online Sexism" (EDOS), aims to address this by not only detecting sexism but also categorizing it into fine-grained classes. A significant hurdle in this task is the severe class imbalance present in the datasets, where the "non-sexist" class overwhelmingly dominates, making it difficult for models to learn to identify the minority "sexist" classes effectively.

Methods

The core of the proposed solution is an ensemble learning approach built upon Transformer-based models. For each of the three subtasks (binary sexism detection, category classification, and fine-grained vector classification), the team trained multiple Transformer models (including BERT, BERTweet, and RoBERTa variants) on various combinations of datasets.

Key methodological aspects include:

  • Data Augmentation: To combat class imbalance, they augmented the original EDOS datasets with data from external sources like EXIST, HatEval, and Multitarget-CONAN. This involved careful data cleaning, translation (using Google Translate API), and synonym replacement to avoid overfitting.
  • Dataset Splitting: For Task A (binary classification), they created four distinct datasets (A1-A4) to explore the impact of different data sources and balancing strategies.
  • Loss Function Adaptation: For tasks with severe imbalance, they experimented with a weighted Binary Cross Entropy loss and Focal Loss to penalize misclassifications of minority classes more heavily.
  • Ensemble Strategies: The final systems for each task were ensembles of the best-performing individual models. They explored both soft voting (averaging probabilities) and hard voting (majority class prediction) ensemble methods.
  • Hyperparameter Tuning: Extensive tuning was performed, particularly for learning rates, to optimize model performance.

Datasets

The paper utilizes several datasets:

  • EDOS (Explainable Detection of Online Sexism) dataset: The primary dataset for the SemEval-2023 Task 10, containing English texts from Reddit and Gab, with labels for three subtasks.
  • EXIST (EXism Identification in Social neTworks) dataset: Used to augment the EDOS data, providing more fine-grained sexism categories.
  • HatEval dataset: Focused on hate speech detection, contributing tweets about women and immigrants.
  • Multitarget-CONAN dataset: Included pairs of hate speech and counter-narratives.
  • MeTwo dataset: Provided Twitter sexism detection data in Spanish.

The authors also mention using provided development data for validation and testing.

Key Insights and Ideas

  • Ensembles Outperform Single Models: Consistently, ensemble models, even with a moderate number of members, achieved better performance than individual models, especially on imbalanced datasets.
  • Data Augmentation's Double-Edged Sword: While adding external data significantly improved performance on minority classes, it could sometimes degrade overall performance or lead to unexpected results, highlighting the importance of careful selection and integration.
  • Task-Specific Strategies are Crucial: The effectiveness of techniques like Focal Loss varied across tasks, indicating that no single solution fits all imbalance scenarios.
  • Fine-grained Categories are Challenging: Task C, with its 11 fine-grained categories, presented the most significant class imbalance and was the most challenging to tackle effectively.
  • Domain-Specific Pre-training Matters: Models pre-trained on domain-specific data (like BERTweet for English tweets or toxicity-specific RoBERTa) generally outperformed general-purpose models.

Key Results

The team achieved competitive results across all three tasks:

  • Task A (Sexism Detection): Ranked 36th out of 84 submissions with a Macro-F1 score of 0.838. A post-competition analysis revealed an ensemble achieving 0.858 on the test set.
  • Task B (Category of Sexism): Ranked 29th out of 69 submissions with a Macro-F1 score of 0.623.
  • Task C (Fine-grained Vector of Sexism): Ranked 25th out of 63 submissions with a Macro-F1 score of 0.457.

These results place them within the top 40% of teams for each track.

Limitations and Future Work

The authors acknowledge several limitations:

  • Computational Resources: Training and deploying ensembles are computationally expensive, requiring significantly more resources than single models.
  • Ensemble Design Complexity: Finding the optimal ensemble composition can be challenging and time-consuming, as the best-performing individual models don't always form the best ensemble.
  • Interpretability of Ensembles: While the task emphasizes explainability, extracting clear, hierarchical explanations from complex ensembles is inherently more difficult than from single models.

Future work directions include:

  • Focusing on Interpretability: Further exploring methods to explain the behavior of ensemble models, potentially by analyzing individual model contributions or attention mechanisms.
  • Mitigating Negative Impacts of Augmentation: Developing strategies to better integrate external data without degrading performance on majority classes.
  • Exploring other Ensemble Strategies: Investigating more advanced ensemble techniques beyond soft and hard voting.

Summary updated

BibTeX
@inproceedings{rydelek2023adamr,
  author = {Adam Rydelek and Daryna Dementieva and Georg Groh},
  title = {{AdamR at SemEval-2023 Task 10: Solving the Class Imbalance Problem in Sexism Detection with Ensemble Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure