DH-FBK at SemEval-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection

Elisa Leonardelli, Camilla CasulaACL 2023

TL;DR

This paper tackles the challenge of accurately and explainably detecting online sexism by proposing a multi-task learning approach that leverages shared representations across different levels of sexism annotation. A key innovation is the artificial estimation and incorporation of inter-annotator agreement as an auxiliary task, demonstrating its value in improving model performance, especially on challenging, ambiguous cases.

What Problem Does This Paper Address?

The internet, while a powerful tool for connection, can also be a breeding ground for harmful content like sexism, which disproportionately affects women and creates hostile online environments. While automated tools exist to flag sexist content, they often lack explainability and struggle with the nuanced, subjective nature of sexism. The SemEval-2023 Task 10 (EDOS) aimed to address this by introducing a fine-grained annotation schema for sexism, moving beyond simple binary classification to include categories and specific vectors of sexist expression. This task presents a significant challenge: how to build models that are not only accurate but also interpretable, and how to handle the inherent subjectivity in human annotation, which can lead to disagreements.

Methods

The DH-FBK team employed a multi-task learning (MTL) approach, a paradigm where a single model learns multiple related tasks simultaneously. The core idea is that by sharing representations, knowledge gained from one task can benefit others. In this case, the three EDOS tasks (binary sexism detection, sexism category, and fine-grained sexism vector) were trained together.

A novel aspect of their approach is the incorporation of inter-annotator agreement as an auxiliary task. Recognizing that disaggregated agreement data wasn't provided, they artificially estimated it. This was done by training an ensemble of five multi-task classifiers. For each fold of the training data, these five classifiers predicted annotations, and the level of agreement among them for a given instance was used as a proxy for inter-annotator agreement. This "ensemble agreement" was then fed into the main multi-task models as an additional input. This strategy is motivated by prior research suggesting that agreement information can be crucial for handling subjective tasks and identifying challenging examples.

For their final submissions, they built two separate models:

  • MT_TASKA: Focused on Task A (binary sexism detection), incorporating ensemble agreement prediction for Task A as an auxiliary task.
  • MT_TASKBC: Designed for Tasks B and C (sexism category and vector), as these tasks are conditional on a post being identified as sexist. This model also included ensemble agreement predictions for Tasks B and C.

Datasets

The primary dataset used is the "Explainable Detection of Online Sexism" (EDOS) dataset, provided by the SemEval-2023 Task 10 organizers. This dataset contains 20,000 social media posts from Gab and Reddit, annotated at three levels:

  1. Binary Sexism Detection (Task A): Sexist or not sexist.
  2. Category of Sexism (Task B): Four categories (threats, derogation, animosity, prejudiced discussions).
  3. Fine-grained Vector of Sexism (Task C): Eleven specific vectors of sexist expression.

The paper also mentions using the gold crowd agreement information released by organizers after the competition for analysis.

Key Insights and Ideas

  • Multi-task learning is effective: Training on related tasks simultaneously significantly improves performance by allowing knowledge sharing.
  • Inter-annotator agreement is a valuable signal: Even artificially estimated agreement can act as a powerful auxiliary task, especially for subjective content.
  • Ambiguity is the enemy: Cases where annotators disagree are demonstrably harder for models to classify, highlighting the importance of understanding disagreement.
  • Ensemble agreement as a proxy: Using the agreement of multiple classifiers can effectively mimic real crowd agreement, making the approach more practical when gold agreement data is unavailable.
  • Hierarchical structure matters: The inherent dependencies between the EDOS tasks (e.g., Task B and C depend on Task A) necessitate careful model design to avoid error propagation.

Key Results

The DH-FBK team achieved competitive results across all three tasks:

  • Task A: Ranked 32nd out of 84 submissions with a macro F1 score of 0.8402.
  • Task B: Ranked 24th out of 69 submissions with a macro F1 score of 0.7326.
  • Task C: Ranked 11th out of 63 submissions with a macro F1 score of 0.5606, their best performance.

The use of multiple restarts (5 models trained with different seeds) and majority voting improved performance, hypothetically advancing their leaderboard positions by 10 for Task A, 4 for Task B, and 11 for Task C. Analysis showed that incorporating agreement information (either artificial or crowd) in training was statistically beneficial compared to not using it. Crucially, performance dropped significantly on test data where annotators disagreed, underscoring the difficulty of these cases.

Limitations and Future Work

The authors acknowledge several limitations:

  • Language Specificity: The current investigation is limited to English and needs expansion to other languages.
  • Subtlety of Sexism: The approach may not fully capture more subtle forms of sexism, which can be overt or implicit.
  • Task B & C Agreement Analysis: The detailed analysis of agreement's role was primarily conducted for Task A and requires further exploration for Tasks B and C.
  • Artificial Agreement Generalizability: While their method reliably estimates crowd agreement, its effectiveness in different circumstances needs further investigation.

Future work could explore the role of agreement in finer-grained labels (Tasks B and C), given the increased complexity and subjectivity. They also highlight the challenge posed by the ~22% of the dataset where annotators failed to reach unanimous consensus, suggesting that incorporating diverse viewpoints is crucial for dataset compilation and classifier design.

Summary updated

BibTeX
@inproceedings{leonardelli2023dh,
  author = {Elisa Leonardelli and Camilla Casula},
  title = {{DH-FBK at SemEval-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure