SINAI at SemEval-2023 Task 10: Leveraging Emotions, Sentiments, and Irony Knowledge for Explainable Detection of Online Sexism

M. Estrella Vallecillo Rodríguez, Flor Miriam Plaza-del-Arco, L. Alfonso Ureña-López, M. Teresa Martín-ValdiviaACL 2023

TL;DR

This paper presents SINAI's approach to the SemEval-2023 Task 10 on detecting online sexism. They demonstrate that incorporating extra-linguistic features like emotions, sentiments, and irony into pre-trained language models significantly improves the detection of nuanced forms of sexism, particularly in multi-class classification scenarios.

What Problem Does This Paper Address?

Online sexism is a pervasive issue, but its detection is challenging due to its varied and often subtle manifestations. Sexist comments can be disguised through irony, mockery, or by embedding prejudiced ideas within seemingly innocuous statements. Traditional binary sexism detection (sexist vs. not sexist) is insufficient, as sexism encompasses a spectrum of behaviors and attitudes. The SemEval-2023 Task 10 aimed to address this by introducing a multi-class classification challenge with three subtasks: binary detection, categorization into four broad classes, and fine-grained classification into eleven specific types of sexism. The core problem is developing models that can not only identify sexism but also understand its underlying mechanisms and nuances, making the detection process explainable.

Methods

The SINAI team developed a system that leverages pre-trained language models, specifically RoBERTa-base and Twitter RoBERTa-base-hate, and augments them with extra-linguistic information. Their core strategy involves extracting features related to emotions, sentiments, and irony from the text using specialized, fine-tuned models. These extracted scores are then concatenated with the output of the main language model (specifically, the [CLS] token's hidden state) and fed into a feed-forward network for final classification. This approach aims to capture the emotional tone, polarity, and potential deceptive language (irony) that often accompany sexist content. They experimented with different combinations of these extra-linguistic features (e.g., emotions + sentiments, emotions + sentiments + irony) to find the optimal configuration for each subtask.

Datasets

The experiments were conducted using the dataset provided by the organizers of the SemEval-2023 Task 10 (EDOS). This dataset consists of posts from Gab and Reddit, annotated for sexism.

  • Training Set: 14,000 posts
  • Development Set: 2,000 posts
  • Test Set: 4,000 posts

The dataset is structured with:

  • Subtask A: Binary classification (sexist/not sexist).
  • Subtask B: Four categories of sexism (threats, derogation, animosity, prejudiced discussions).
  • Subtask C: Eleven fine-grained classes of sexism (e.g., descriptive attacks, dehumanizing attacks, gender stereotypes, condescending explanations).

Key Insights and Ideas

  • The Power of Nuance: Simply detecting "sexist" or "not sexist" is insufficient. The paper strongly suggests that understanding the type and manner of sexism (e.g., through emotions, irony) is crucial for accurate detection, especially in more complex scenarios.
  • Emotions and Irony as Key Indicators: The study highlights that emotions (particularly negative ones like disgust and anger) and irony are significant signals that can help identify and mask sexist content. Incorporating these features consistently improved performance, especially in the more granular subtasks.
  • Twitter-Specific Models Shine: The Twitter RoBERTa-base-hate model, pre-trained on tweets and fine-tuned for hate speech, generally outperformed the generic RoBERTa-base model, particularly in subtask C. This indicates that domain-specific pre-training can be highly beneficial for tasks involving social media language.
  • Diminishing Returns for Simple Combinations: While adding emotions and sentiments improved performance in some cases, adding irony often provided the most significant boost, especially for the fine-grained subtask C. The combination of all three features (emotions + sentiments + irony) proved most effective for the most challenging subtask.

Key Results

The SINAI team achieved the following rankings and macro F1-scores:

  • Subtask A (Binary Sexism Detection): Ranked 46th with a macro F1-score of 0.8245. The best performing model for this subtask was Twitter RoBERTa-base-hate with emotions + sentiments.
  • Subtask B (Category of Sexism): Ranked 37th with a macro F1-score of 0.6043. The baseline RoBERTa-base model performed best here, suggesting that the added features did not significantly improve performance for these broader categories.
  • Subtask C (Fine-grained Vector of Sexism): Ranked 29th with a macro F1-score of 0.4376. The Twitter RoBERTa-base-hate model with emotions + sentiments + irony achieved the best results for this subtask.

Limitations and Future Work

  • Subtask B Performance: The authors noted that for subtask B, the baseline RoBERTa-base model outperformed those that incorporated extra-linguistic features, contrary to their initial hypothesis. This suggests that the broader categories in subtask B might not benefit as much from these specific features compared to the fine-grained distinctions in subtask C.
  • Unbalanced Data: In their error analysis for subtask C, the authors observed that their model never predicted the "condescending explanations or unwelcome advice" label. They attribute this to the imbalanced nature of the dataset, where this specific class has fewer examples.
  • Future Work: The team plans to explore other methods for incorporating extra-linguistic information into pre-trained language models. They also intend to investigate the impact of different model architectures on the performance when incorporating these features.

Summary updated

BibTeX
@inproceedings{rodriguez2023sinai,
  author = {M. Estrella Vallecillo Rodríguez and Flor Miriam Plaza-del-Arco and L. Alfonso Ureña-López and M. Teresa Martín-Valdivia},
  title = {{SINAI at SemEval-2023 Task 10: Leveraging Emotions, Sentiments, and Irony Knowledge for Explainable Detection of Online Sexism}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure