HULAT at SemEval-2023 Task 10: Data Augmentation for Pre-trained Transformers Applied to the Detection of Sexism in Social Media

Isabel Segura-BedmarACL 2023

TL;DR

This paper explores using pre-trained transformer models (like BERT and RoBERTa) to detect sexism in social media posts. They found that while data augmentation didn't significantly improve performance on the binary sexism detection task, it showed promise for more nuanced, fine-grained classification, particularly with RoBERTa.

What Problem Does This Paper Address?

Sexism, unfortunately, remains prevalent online, with social media platforms often used to spread hateful and sexist messages. Automatically detecting this harmful content is crucial, but existing methods often provide a high-level classification without explaining why a post is considered sexist. SemEval-2023 Task 10 aimed to address this by promoting the development of explainable and fine-grained models for detecting sexism in English social media posts from Gab and Reddit. The task was broken down into three subtasks: (A) binary sexism detection, (B) a four-class categorization of sexism, and (C) an eleven-class fine-grained vector of sexism.

Methods

The researchers experimented with several popular pre-trained transformer models, including BERT, DistilBERT, RoBERTa, and XLNet. To address the challenge of limited training data, they employed data augmentation techniques. Specifically, they used EDA (Easy Data Augmentation) and the NLPAug library to generate synthetic data by applying operations like synonym replacement, random insertion, random swap, and random deletion. These augmented datasets were then used to fine-tune the transformer models. The core of their approach involved fine-tuning these models on the provided datasets for each of the three subtasks.

Datasets

The primary dataset for this task was collected from Gab and Reddit, comprising 20,000 English posts. This dataset was split into training (14,000 posts), development (2,000 posts), and final evaluation (4,000 posts) sets. The dataset was annotated for three subtasks:

  • Task A: Binary classification (sexist vs. non-sexist).
  • Task B: Four categories of sexism (threats, derogation, animosity, prejudiced discussions).
  • Task C: Eleven fine-grained vectors of sexism.

The paper notes that the dataset exhibits class imbalance, particularly in Task A where the non-sexist class is dominant.

Key Insights and Ideas

  • RoBERTa Shines: RoBERTa consistently emerged as the top-performing model across the tasks, often outperforming BERT and its variants.
  • Data Augmentation's Mixed Bag: The impact of data augmentation was task-dependent. It did not significantly improve results for the binary classification (Task A) but showed positive effects for the more granular tasks (B and C), especially when combined with RoBERTa.
  • Fine-Grained is Tough: The fine-grained classification tasks (B and C) proved significantly more challenging, with lower overall F1-scores compared to Task A. This highlights the difficulty in capturing the nuances of different sexist expressions.
  • Text Length Matters (Slightly): While not a primary focus, the paper observed some correlations between text length and specific categories or vectors of sexism, suggesting that linguistic features can play a role. For instance, "prejudiced discussions" tended to be longer, while "threats" were shorter and often contained more violent vocabulary.
  • Synthetic Data Trade-offs: While synthetic data can boost recall, it can sometimes negatively impact precision, as seen with RoBERTa in Task B. This suggests a careful balance is needed when generating augmented data.

Key Results

The HULAT team achieved the following rankings and Macro F1-scores:

  • Task A (Binary Sexism Detection): 43rd out of 84 teams with a Macro F1-score of 0.8298. RoBERTa (without data augmentation) was used for submission.
  • Task B (Category of Sexism): 45th out of 69 teams with a Macro F1-score of 0.5877. RoBERTa with data augmentation was used for submission.
  • Task C (Fine-grained Vector of Sexism): 27th out of 63 teams with a Macro F1-score of 0.4458. RoBERTa with data augmentation was used for submission.

Notably, for Task C, data augmentation was crucial, as RoBERTa without it struggled to classify certain vectors.

Limitations and Future Work

The authors acknowledge that their approach has "much room for improvement," particularly in the fine-grained classification tasks. They also note that data augmentation did not consistently improve results across all tasks and models.

Future work planned includes:

  • Exploring more data augmentation techniques, such as back-translation.
  • Investigating other datasets for sexist content detection, including multilingual and multimodal scenarios.
  • Expanding the approach to handle different scenarios beyond the current task.

Summary updated

BibTeX
@inproceedings{segurabedmar2023hulat,
  author = {Isabel Segura-Bedmar},
  title = {{HULAT at SemEval-2023 Task 10: Data Augmentation for Pre-trained Transformers Applied to the Detection of Sexism in Social Media}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure