LCT-1 at SemEval-2023 Task 10: Pre-training and Multi-task Learning for Sexism Detection and Classification

Konstantin Chernyshev, Ekaterina Garanina, Duygu Bayram, Qiankun Zheng, Lukas EdmanACL 2023

TL;DR

This paper tackles the challenge of detecting and classifying online sexism, focusing on improving model interpretability. The authors explore domain-adaptive pre-training and multi-task learning, finding that while pre-training generally boosts performance, the effectiveness of multi-task learning varies significantly across different levels of sexism classification granularity.

What Problem Does This Paper Address?

Online sexism is a pervasive issue with significant negative impacts. While significant progress has been made in detecting online sexism using machine learning, many existing systems lack interpretability, making it difficult to understand why a piece of text is flagged as sexist. SemEval-2023 Task 10 (EDOS) aims to address this by focusing on explainable detection of online sexism. This involves not only flagging sexist content but also classifying the specific type of sexism present, ranging from broad categories to very fine-grained distinctions. The authors' work contributes to this effort by investigating how advanced pre-training and multi-task learning techniques can enhance the performance of sexism detection models, particularly in the context of limited annotated data.

Methods

The core of the authors' approach involves leveraging pre-trained Transformer models and enhancing them through two primary strategies:

  • Domain-Adaptive Pre-training (DAPT) and Task-Adaptive Pre-training (TAPT): Building on the idea that pre-training language models on domain-specific data can improve downstream task performance, the authors explore two methods. DAPT involves further pre-training on a large corpus of domain-related text (2 million texts from Reddit and Gab, plus other hate speech data). TAPT, on the other hand, uses only the target task data for unsupervised pre-training. They also experiment with a sequential DAPT+TAPT approach.
  • Multi-Task Learning (MTL): To leverage available annotated data from related tasks (hate speech, sexism, misogyny), the authors employ MTL. This approach trains a single model to perform multiple tasks simultaneously, sharing a common encoder and using task-specific heads. They hypothesize that MTL can enrich the model's understanding by exposing it to a wider variety of related linguistic phenomena, especially when target task data is scarce. They utilize the MaChAmp toolkit for their MTL experiments.

Preprocessing steps were also investigated, including uniform masking of usernames and URLs, emoji normalization, and hashtag segmentation.

Datasets

The authors utilized several datasets for their experiments:

  • EDOS Dataset: The primary dataset for the SemEval-2023 Task 10, consisting of 14,000 messages from Reddit and Gab, annotated at three levels of granularity:
  • Task A: Binary classification (sexist/not sexist).
  • Task B: 4-category sexism classification.
  • Task C: 11-category sexism classification. This dataset also includes 2 million unlabeled texts.

  • Related Datasets for Pre-training and MTL: A collection of publicly available datasets for hate speech (HS), sexism, and misogyny detection were used. These include:

  • HS Datasets: OLID, HatEval, Measuring HS, UB, HaSpeeDe.
  • Sexism/Misogyny Datasets: AMI@EVALITA2018, Call me sexist, EXIST, Online misogyny.

Key Insights and Ideas

  • Domain-Adaptive Pre-training is Generally Beneficial: Further pre-training on domain-specific data (DAPT) consistently improved performance on the sexism detection and classification tasks compared to standard fine-tuning.
  • MTL's Effectiveness is Task-Dependent: Multi-task learning showed significant benefits for Task B (4-category sexism classification), performing better than standard fine-tuning. However, for Task A (binary sexism detection), MTL performed on par with standard fine-tuning, and for Task C (fine-grained 11-category classification), standard fine-tuning was preferable. This highlights that the complexity and granularity of the task heavily influence the optimal training strategy.
  • Task Formulation Matters: The way tasks are defined and grouped for MTL significantly impacts performance. For Task A, incorporating related tasks like general hate speech and offensive language detection proved beneficial, suggesting that broader linguistic understanding can aid specific sexism detection.
  • Preprocessing Choices Impact Performance: While the paper explored various preprocessing steps, the most effective combination for their experiments involved mask normalization and emoji normalization, with hashtags left intact.

Key Results

  • Pre-training: Domain-adaptive pre-training (DAPT) using 2 million texts led to improved F1-macro scores across all tasks compared to baseline models. Task-adaptive pre-training (TAPT) also showed improvements but generally yielded lower scores than DAPT.
  • MTL:
    • For Task A, MTL on related HS datasets achieved comparable results to standard fine-tuning.
  • For Task B, MTL on sexism datasets improved performance over standard fine-tuning.
  • For Task C, standard fine-tuning consistently outperformed MTL.
  • Multilingual Experiments: Exploring multilingual MTL showed a noticeable drop in performance, leading the authors to discontinue this line of research.

Limitations and Future Work

  • Partition Unevenness: The authors observed a significant score decrease on the test set compared to the development set, attributing this to partition unevenness and the complexity of the task. This made model selection challenging.
  • MTL Complexity: While MTL showed promise, the impact of adding various external datasets was sometimes arbitrary, making it difficult to draw definitive conclusions about optimal dataset combinations.
  • Future Work: The paper suggests that further domain adaptation pre-training is a promising avenue for improving sexism detection and classification. The selection of fine-tuning methods should be carefully considered based on the specific task. Future work could explore more sophisticated MTL strategies and investigate the reasons behind the observed partition unevenness.

Summary updated

BibTeX
@inproceedings{chernyshev2023lct,
  author = {Konstantin Chernyshev and Ekaterina Garanina and Duygu Bayram and Qiankun Zheng and Lukas Edman},
  title = {{LCT-1 at SemEval-2023 Task 10: Pre-training and Multi-task Learning for Sexism Detection and Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure