JRLV at SemEval-2022 Task 5: The Importance of Visual Elements for Misogyny Identification in Memes

Jason Ravagli, Lorenzo VaianiNAACL 2022

TL;DR

This paper tackles the challenge of automatically identifying misogynistic memes by leveraging both visual and textual information. Their approach, combining Mask R-CNN for image feature extraction and VisualBERT for multimodal understanding, achieved a competitive F1 score, highlighting the crucial role of visual elements, particularly people, in detecting misogyny.

What Problem Does This Paper Address?

Misogyny and gender discrimination are pervasive issues on social media, and memes have become a popular medium for spreading such harmful content. Manually moderating the vast amount of online content is infeasible, necessitating automated tools. The SemEval-2022 Multimedia Automatic Misogyny Identification (MAMI) task, specifically subtask A, aims to classify memes as either misogynous or non-misogynous. This is a challenging problem because the offensive message in a meme can be conveyed through a complex interplay of images and text, where one modality might be neutral or even misleading on its own. The authors aim to build a system that can effectively fuse information from both modalities to accurately identify these subtle forms of hate speech.

Methods

The core of the proposed solution lies in a multimodal approach that leverages state-of-the-art deep learning models.

  • Image Feature Extraction: Mask R-CNN is employed to identify and extract relevant "regions of interest" (ROIs) from meme images. The authors experiment with two pre-training strategies for Mask R-CNN: one on the COCO dataset (91 classes) and another on the LVIS dataset (1230 classes). They observe that COCO pre-training is more effective at detecting "person" objects, which are prevalent in their analysis.
  • Multimodal Fusion: VisualBERT, a transformer-based model designed for vision-and-language tasks, is used to process both text and image features. VisualBERT's self-attention mechanism allows it to align elements from text and image regions, effectively creating an early fusion point. This means that the model learns to correlate visual and textual information from the outset, rather than processing them separately and combining them later.
  • Text Encoding: Standard BERT is used to generate text embeddings.
  • Classification: The combined multimodal representation from VisualBERT is fed into a fully connected layer for binary classification (misogynous/non-misogynous).

The authors also explore unimodal baselines (text-only and image-only) to understand the contribution of each modality and compare different fusion strategies.

Datasets

The primary dataset used is the one provided for the SemEval-2022 Task 5 (MAMI).

  • Training Data: 10,000 memes, each consisting of an image and its corresponding text transcript. These memes are labeled with five binary tags, including misogyny, stereotype, shaming, objectification, and violence. For their experiments, the authors reserve 25% of this data for validation.
  • Test Data: 1,000 memes, with labels released only after the competition.

The Mask R-CNN models were pre-trained on the COCO and LVIS datasets.

Key Insights and Ideas

  • Visual Elements are Crucial: The paper strongly emphasizes that visual elements, particularly the presence of people, play a significant role in identifying misogynous memes. Their model's performance significantly improves when visual information is incorporated.
  • Early Fusion is Key: The success of VisualBERT, which inherently performs early fusion, suggests that integrating visual and textual information from the beginning is more effective than late fusion for this task. This allows the model to build a richer contextual understanding.
  • "Person" Detection is a Strong Indicator: The COCO pre-training for Mask R-CNN, which excels at detecting "person" objects, led to better performance. This indicates that misogynous memes often target or depict people, making their identification a critical feature.
  • Not All Visual Information is Equal: The authors found that simply using the entire image with a general CNN like VGG16 was insufficient. Focusing on specific detected regions (ROIs) using Mask R-CNN proved more effective, especially when those regions contained people.
  • Subtle Cases Require Multimodality: The paper highlights that some memes have a neutral image with offensive text, or vice-versa. These cases are difficult to classify with unimodal approaches and necessitate a multimodal solution.

Key Results

  • The proposed VisualBERT-based solution achieved an F1 score of 0.67 on the test set, which was 2% higher than the best baseline provided by the task organizers.
  • Unimodal BERT performed better than unimodal VGG16, indicating that text is a strong indicator for misogyny.
  • The best performing VisualBERT model used the CLS token embedding for classification.
  • Using both COCO and LVIS pretraining for Mask R-CNN jointly did not yield further improvements, likely due to the high overlap in detected regions and the dominance of "person" detection from COCO.

Limitations and Future Work

  • Dataset Size and Overfitting: The authors acknowledge that the dataset is relatively small, leading to overfitting issues during training. They experimented with various strategies (e.g., pre-training on the Hateful Memes task, freezing layers) but did not achieve significant improvements in generalization.
  • Image Information Extraction: A major obstacle identified is the extraction of meaningful information from images. The authors plan to enrich their preprocessing by incorporating image captioning and entity detection to convert visual information into a textual format, further enhancing the multimodal understanding.
  • Investigating Visual Elements: Future work will focus on a deeper investigation into the specific visual elements that contribute to misogyny detection and how their usage impacts performance.

Summary updated

BibTeX
@inproceedings{ravagli2022jrlv,
  author = {Jason Ravagli and Lorenzo Vaiani},
  title = {{JRLV at SemEval-2022 Task 5: The Importance of Visual Elements for Misogyny Identification in Memes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure