IUST_NLP at SemEval-2023 Task 10: Explainable Detecting Sexism with Transformers and Task-adaptive Pretraining

Hadiseh MahmoudiACL 2023

TL;DR

This paper presents a system for detecting and classifying online sexism using transformer-based models and task-adaptive pretraining. The authors achieved strong results on the SemEval-2023 Task 10, demonstrating the effectiveness of their ensemble and fine-tuning strategies for this challenging NLP task.

What Problem Does This Paper Address?

Online sexism, including discrimination and misogyny, poses a significant harm in social networks. Identifying and classifying this harmful content is complex due to the diverse forms it can take. The SemEval-2023 Task 10 (EDOS) aims to develop accurate and explainable English-language models for detecting sexism across different platforms like Gab and Reddit. The task is structured into three hierarchical subtasks: binary sexism detection (sexist vs. non-sexist), categorization of sexism (threats, derogation, animosity, prejudiced discussions), and fine-grained vector classification of sexism (11 specific categories). This work tackles the challenge of building robust models for these subtasks, especially given the often imbalanced nature of the data and the nuances of sexist language.

Methods

The core of the system relies on transformer-based models, a standard in modern NLP. The authors experimented with several pre-trained models, including BERT, RoBERTa, and DeBERTa. To adapt these general models to the specific domain of online sexism, they employed two key strategies:

  • Task-Adaptive Pretraining: They further pre-trained a BERT model on a large, unlabeled dataset from the same domain (Gab and Reddit). This helps the model learn domain-specific language patterns before being fine-tuned on the labeled data.
  • Ensemble Learning: They combined the predictions of multiple transformer models (DeBERTa, RoBERTa, ERNIE) to leverage their diverse strengths. This ensemble approach involved feeding the input to each model, extracting their representations, adding dropout layers, and then concatenating these features before passing them through dense layers for classification.

For all subtasks, they used a cross-entropy loss function for multi-class classification. To address data imbalance, they experimented with class weights in the loss function, which proved beneficial for subtasks B and C.

Datasets

The primary datasets used were provided by SemEval-2023 Task 10, comprising data from Reddit and Gab. The dataset includes:

  • Labeled Data: Approximately 20,000 labeled instances across the three subtasks. Specifically, Subtask A has 14,000 training instances, while Subtasks B and C have 3,398 training instances each. Data for subtasks B and C also includes samples labeled as sexist in subtask A.
  • Unlabeled Data: A substantial corpus of two million unlabeled instances from the same domain, used for task-adaptive pretraining.

The data was split into training, development (dev), and test sets, with 70% for training, 10% for dev, and 20% for testing.

Key Insights and Ideas

  • Domain Adaptation is Crucial: General-purpose transformer models, while powerful, benefit significantly from domain-specific pretraining (task-adaptive pretraining) to perform well on specialized tasks like sexism detection.
  • Ensembling Boosts Performance: Combining multiple transformer models, even with their individual limitations, can lead to improved robustness and accuracy, especially when dealing with complex classification tasks.
  • Class Weights Mitigate Imbalance: For imbalanced datasets, incorporating class weights into the loss function is an effective strategy to prevent models from being biased towards the majority class and to improve performance on minority classes, particularly for subtasks B and C.
  • Subtask Difficulty Varies: The results highlight that detecting binary sexism (Subtask A) is more straightforward than classifying specific categories of sexism (Subtask B) or fine-grained vectors (Subtask C), which are significantly more challenging due to data scarcity and class granularity.

Key Results

The system achieved the following Macro-F1 scores on the test dataset:

  • Subtask A (Binary Sexism Detection): 83%
  • Subtask B (Category of Sexism): 64%
  • Subtask C (Fine-grained Vector of Sexism): 47%

These results were competitive, with the DeBERTa model (with class weights) showing strong performance, especially on Subtask A. The ensemble model also performed well, particularly on Subtask A. The authors noted that while class weights improved performance, the improvement was more pronounced for Subtasks B and C.

Limitations and Future Work

  • Data Scarcity for Complex Subtasks: The limited amount of labeled data for Subtasks B and C, coupled with the higher number of classes, presents a significant challenge for achieving higher performance.
  • Model Size: The authors suggest that using larger pre-trained models, such as DeBERTa-v3, could potentially lead to better results.
  • Unsupervised Methods: Exploring other unsupervised methods for pretraining or task adaptation could further enhance the system's capabilities.
  • More Data: Acquiring and utilizing more labeled data, especially for the more granular subtasks, is a clear path for future improvement.

Summary updated

BibTeX
@inproceedings{mahmoudi2023iust,
  author = {Hadiseh Mahmoudi},
  title = {{IUST\_NLP at SemEval-2023 Task 10: Explainable Detecting Sexism with Transformers and Task-adaptive Pretraining}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure