FiRC at SemEval-2023 Task 10: Fine-grained Classification of Online Sexism Content Using DeBERTa

Fadi Hassan, Abdessalam Bouchekif, Walid AransaACL 2023

TL;DR

This paper presents a system for detecting and classifying online sexism, achieving top rankings in the SemEval-2023 Task 10. The approach leverages fine-tuned DeBERTa models with k-fold cross-validation, demonstrating the effectiveness of ensemble methods and robust pre-trained models for this challenging NLP task.

What Problem Does This Paper Address?

Online platforms are increasingly grappling with the proliferation of toxic content, including sexism, which creates hostile environments. Identifying and understanding sexism is crucial for content moderation and fostering healthier online spaces. The SemEval-2023 Task 10 specifically aimed to advance research in "Explainable Detection of Online Sexism," focusing on fine-grained classification of sexist content from social media platforms like Gab and Reddit. This task involved not only identifying sexist content but also explaining why it's sexist through hierarchical categorization. The challenge lies in accurately classifying subtle forms of sexism and dealing with potentially imbalanced datasets.

Methods

The core of the proposed system is the DeBERTa (Decoding-enhanced BERT with disentangled attention) model, a powerful transformer architecture that improves upon BERT and RoBERTa by incorporating novel techniques like disentangled attention and an enhanced mask decoder. To address the challenge of limited training data and prevent overfitting, the authors employed k-fold cross-validation. This involved dividing the training data into k subsets, fine-tuning a DeBERTa model on k-1 subsets, and using the remaining subset for validation. The final system was an ensemble of 10 such models, with their predictions aggregated using a soft voting approach. This ensemble strategy, combining models of different sizes (large, xlarge, xxlarge), proved more effective than using a single model. The system was designed to handle three hierarchical subtasks: binary sexism detection (Task A), category of sexism (Task B), and fine-grained vector of sexism (Task C).

Datasets

The research utilized datasets provided by the SemEval-2023 Task 10 organizers. These datasets were primarily in English and included labeled and unlabeled data. The training data for Task A consisted of approximately 14,000 samples, with a development set of 2,000 and a test set of 4,000. Tasks B and C focused on classifying positive "sexist" samples, with significantly fewer labeled instances compared to Task A. The authors noted that the class distribution in the training data was imbalanced, with a much higher proportion of "not sexist" samples, which posed a challenge for model training.

Key Insights and Ideas

  • DeBERTa's Superiority: The study confirmed that DeBERTa models, particularly larger variants, consistently outperformed other transformer architectures like BERT and RoBERTa for this task.
  • Ensembling for Robustness: Combining multiple fine-tuned DeBERTa models of varying sizes through k-fold cross-validation significantly improved performance compared to single models, highlighting the benefits of ensemble learning for robustness and generalization.
  • Monolingual Models Shine: For English-only tasks, monolingual pre-trained models demonstrated better performance than multilingual ones, suggesting that specialized models are more effective when language is not a barrier.
  • K-fold Cross-Validation's Impact: The k-fold cross-validation approach was crucial in mitigating the effects of limited training data, leading to more stable and higher performance scores, especially for classes with fewer examples.
  • Explainability as a Feature: The hierarchical nature of the task (Task A -> Task B -> Task C) provided a data-driven pathway for explainability, allowing the model to offer detailed insights into the sexist violation beyond a simple binary classification.

Key Results

The proposed system achieved strong results in the SemEval-2023 Task 10, ranking:

  • 2nd in Task A (Binary Sexism Detection)
  • 7th in Task B (Category of Sexism)
  • 4th in Task C (Fine-grained Vector of Sexism)

The submitted model achieved an F1-score of 87.4% on the test set for Task A, 70.58% for Task B, and 54.04% for Task C. The analysis of confusion matrices revealed specific areas where the model struggled, such as distinguishing between certain fine-grained categories in Task B and C, often due to semantic similarities between classes.

Limitations and Future Work

The authors acknowledge the imbalanced nature of the training data as a significant limitation, particularly for Task A, which led to lower recall for the "sexist" class. They suggest potential solutions like adding more positive samples or employing data augmentation techniques.

For future work, the authors plan to:

  • Improve the model by using monolingual data and data augmentation techniques to address the limited training data for certain classes.
  • Further explore multilingual models if the task were extended to include other languages.
  • Investigate more sophisticated ensemble strategies beyond soft voting.
  • Continue to leverage the explainability aspect of the task to provide deeper insights into model decisions.

Summary updated

BibTeX
@inproceedings{hassan2023firc,
  author = {Fadi Hassan and Abdessalam Bouchekif and Walid Aransa},
  title = {{FiRC at SemEval-2023 Task 10: Fine-grained Classification of Online Sexism Content Using DeBERTa}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure