MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection

Yimeng Gu, Ignacio Castro, Gareth TysonNAACL 2022

TL;DR

This paper introduces MMVAE, a novel multi-modal, multi-task Variational Autoencoder designed to automatically detect misogynous memes. By effectively co-representing visual and textual features, MMVAE achieves state-of-the-art performance on the SemEval-2022 MAMI task, outperforming unimodal baselines and offering a more nuanced understanding of harmful content.

What Problem Does This Paper Address?

Memes have become a ubiquitous form of online communication, often used for humor and expression. However, their easy composition also allows for the embedding of malicious content, including misogyny, which can be harmful to targeted groups. Traditional text-based moderation tools struggle to effectively identify such harmful content within memes, as the meaning is often derived from the interplay between images and text, which can be subtle or even contradictory. This paper addresses the challenge of automatically detecting misogynous memes by developing a model that can understand and integrate information from both visual and textual modalities. Furthermore, it aims to go beyond simple binary classification by identifying fine-grained categories of misogyny, such as shaming, stereotype, objectification, and violence.

Methods

The core of the proposed solution is the Multi-modal Multi-task Variational AutoEncoder (MMVAE). This model is built upon three key components:

  • Image/Text Embedding Module: This module leverages pre-trained models to generate rich vector representations for both the meme's image and its transcribed text. The authors experimented with various pre-trained models for both modalities, including ResNet-50 and CLIP for images, and BERT, LASER, LaBSE, and CLIP for text.
  • Variational AutoEncoder (VAE) Module: This module is responsible for fusing the uni-modal embeddings into a shared latent space, creating a multi-modal co-representation. The VAE learns to reconstruct the original embeddings from this latent representation, effectively capturing the inter-relationship between the visual and textual features. The goal is to learn a latent variable that encapsulates the semantic meaning of the meme.
  • Multi-Task Learning Module: This component takes the co-representation from the VAE and uses it to predict labels for multiple tasks simultaneously. Specifically, it performs binary classification for misogyny detection (Sub-task A) and multi-class classification for fine-grained misogyny categories (shaming, stereotype, objectification, violence) (Sub-task B). The multi-task learning approach is expected to improve performance by allowing the model to learn from related tasks and share knowledge.

The overall training objective is a weighted sum of the reconstruction losses from the VAE, the KL divergence term (standard in VAEs), and the cross-entropy losses from each of the multi-task classification objectives.

Datasets

The primary dataset used in this work is the Multimedia Automatic Misogyny Identification (MAMI) dataset, released for the SemEval-2022 Task 5. This dataset contains 10,000 memes for training and 1,000 for testing, with each meme having an associated text transcription and labels for five classes: misogyny, shaming, stereotype, objectification, and violence. The paper notes that the label distributions for the fine-grained categories are imbalanced, with only the misogyny class being balanced.

Key Insights and Ideas

  • The power of co-representation: The paper strongly emphasizes that understanding misogynous memes requires integrating both visual and textual information. Standalone analysis of either modality is often insufficient, as the harmful meaning can emerge from their interaction.
  • VAE for fusion: Using a Variational Autoencoder for fusing multi-modal representations is a key methodological choice. This allows for learning a probabilistic latent space that can capture complex relationships and generate a robust co-representation.
  • Multi-task learning for nuance: Applying multi-task learning to predict both the binary misogyny label and specific sub-categories (shaming, etc.) is crucial. This allows the model to learn more nuanced distinctions and potentially improve overall performance by leveraging shared information across related tasks.
  • Pre-trained models are essential: The success of the MMVAE heavily relies on the effective use of pre-trained models for initial feature extraction, demonstrating the importance of transfer learning in this domain.

Key Results

The proposed MMVAE model achieved strong performance on the SemEval-2022 MAMI task:

  • Sub-task A (Misogyny Detection): The best performing MMVAE variant achieved an F₁ score of 0.723, significantly outperforming the text-only BERT baseline by 22.8% and the image-only ResNet-50 baseline by 16.2%.
  • Sub-task B (Fine-grained Classification): The optimal F₁ score for this sub-task was 0.634.
  • Error Analysis: The model was found to be better at identifying misogynous memes with high precision but less effective at correctly labeling non-misogynous memes (higher false positive rate). Analysis of false positives suggested that issues can arise from text transcriptions that include crossed-out hateful content or words associated with femininity, even if the meme itself isn't explicitly misogynous.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Gap between validation and test performance: A significant drop in performance was observed between the validation and test sets, indicating potential overfitting or dataset shift.
  • Impact of text transcription: The current text transcription method might include irrelevant or misleading information (e.g., crossed-out text), which can confuse the classifier.
  • Subtle misogyny and word associations: The model struggles with certain types of false positives, particularly when hateful text is present but not the primary focus, or when words associated with femininity are present.
  • Future work: The authors plan to investigate the impact of text transcription methods further, explore the influence of words associated with femininity, and integrate attention mechanisms into the model for more sophisticated co-representation. They also aim to conduct more comprehensive statistical analyses of model results.

Summary updated

BibTeX
@inproceedings{gu2022mmvae,
  author = {Yimeng Gu and Ignacio Castro and Gareth Tyson},
  title = {{MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure