SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification

Elisabetta Fersini, Francesca Gasparini, Giulia Rizzi, Aurora Saibene, Berta Chulvi, Paolo Rosso, Alyssa Lees, Jeffrey SorensenNAACL 2022

TL;DR

This paper introduces SemEval-2022 Task 5 (MAMI), a challenge focused on automatically identifying misogynous memes by analyzing both text and images. The task saw significant participation, highlighting the growing interest in detecting harmful content in multimodal online formats and revealing the complexities of this task for AI systems.

What Problem Does This Paper Address?

The internet, particularly image-based social media, has become a significant platform for communication. However, alongside opportunities, it also hosts a proliferation of offensive content, including misogynistic material. Memes, a popular communication tool combining images and text, are frequently used to spread sexist and aggressive messages, amplifying offline inequalities. While previous work has tackled misogyny detection, it often relied solely on text. This paper addresses the need for a multimodal approach to identify misogynous memes, recognizing that the harmful intent can be subtle and implicit, requiring the analysis of both visual and textual components.

Methods

The core of this work is the establishment of SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification (MAMI). The task was structured into two sub-tasks:

  • Sub-task A: A binary classification task to determine if a meme is misogynous or not.
  • Sub-task B: A multi-label classification task to identify the specific type of misogyny present, categorizing memes into shaming, stereotype, objectification, and violence.

The MAMI challenge involved participants developing systems to tackle these tasks. The paper also describes the dataset creation process, which involved collecting memes from social media platforms and meme creation websites, followed by human annotation.

Datasets

The paper describes the creation of a new benchmark dataset for multimedia misogyny identification.

  • Training Set: 10,000 memes.
  • Testing Set: 1,000 memes.

The dataset was annotated by human observers, with labels for misogynous/non-misogynous and specific types of misogyny (shaming, stereotype, objectification, violence). Inter-annotator agreement was measured using Fleiss's kappa, indicating moderate agreement for the binary task and fair agreement for the type classification task.

Key Insights and Ideas

  • Multimodality is Crucial: The task's success and the challenges faced by participants underscore that misogyny in memes is often conveyed through the interplay of text and images, making unimodal approaches insufficient.
  • Subtlety of Misogyny: Many misogynous memes are not overtly aggressive, making them difficult to detect. The implicit nature of sexism, particularly in forms like "benevolent sexism," poses a significant challenge for AI.
  • Bias in Models: The observation that models tend to misclassify "not misogynous" memes as misogynous more often than the reverse suggests a potential bias in current systems, possibly leaning towards over-detection of misogyny.
  • Type Classification is Harder: Sub-task B, identifying the specific type of misogyny, proved more challenging than the binary classification, as indicated by lower agreement scores and performance metrics.

Key Results

The MAMI challenge attracted a large number of participants: 65 teams for Sub-task A and 41 for Sub-task B.

  • Sub-task A Performance: The top-performing systems achieved a macro-average F1-score of 0.834. Many teams (72%) outperformed the provided baseline models.
  • Sub-task B Performance: The highest weighted-average F1-score achieved was 0.731.
  • Error Analysis: The analysis revealed that misclassifying "not misogynous" memes as misogynous (False Positives) was a common issue, particularly when memes contained elements that could be misinterpreted. Conversely, missing actual misogynous memes (False Negatives) was also prevalent, especially in cases of subtle objectification or violence.

Limitations and Future Work

  • Model Bias: The paper highlights the tendency for models to be biased towards detecting misogyny, leading to a higher rate of false positives for non-misogynous content. Future work should focus on developing less biased models.
  • Subtlety and Implicit Sexism: The challenge of detecting subtle forms of misogyny, such as benevolent sexism or implicit sexual objectification, remains an open area.
  • Dynamic Nature of Memes: The constant evolution of memes with new images and text presents an ongoing challenge for systems to keep pace.
  • Annotation Subjectivity: The paper acknowledges that some categories, like "stereotype," can be subjective and influenced by societal views, as seen with the discussion on benevolent sexism. Future research could explore more robust annotation strategies or consider different interpretations.

Summary updated

BibTeX
@inproceedings{fersini2022semeval,
  author = {Elisabetta Fersini and Francesca Gasparini and Giulia Rizzi and Aurora Saibene and Berta Chulvi and Paolo Rosso and Alyssa Lees and Jeffrey Sorensen},
  title = {{SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure