Mitra Behzadi at SemEval-2022 Task 5 : Multimedia Automatic Misogyny Identification method based on CLIP

Mitra Behzadi, Ali Derakhshan, Ian HarrisNAACL 2022

TL;DR

This paper tackles the challenge of automatically identifying misogynistic memes, which combine images and text to spread hateful content. The authors propose a novel multi-modal approach using the CLIP network to not only detect misogynistic memes but also classify them into specific subcategories like shaming, stereotype, objectification, or violence, achieving competitive results in the SemEval-2022 Task 5 challenge.

What Problem Does This Paper Address?

The proliferation of social media has led to the widespread use of memes as a communication tool. While many memes are harmless, a significant portion can be deeply offensive and hateful, particularly towards women. Misogynistic memes often leverage the interplay between visual and textual elements, making them challenging to detect using traditional text-only methods. The visual aspect can obscure or complement the text, creating a complex communication layer that requires a multi-modal understanding. This paper addresses the critical need for automated systems capable of identifying these harmful memes and understanding the specific forms of misogyny they employ.

Methods

The core of the proposed method is the use of CLIP (Contrastive Language-Image Pre-training), a powerful multi-modal network initially designed for object detection. The authors adapt CLIP for misogyny identification by treating it as a multi-label classification problem.

Here's a breakdown of their approach:

  • Multi-modal Encoding: CLIP is used to independently encode both the image and text components of a meme. This generates rich feature representations for each modality.
  • Feature Concatenation: The image and text features are then concatenated, creating a combined representation that captures the essence of the meme.
  • Feed-Forward Network: A feed-forward neural network is employed to learn a non-linear function that maps these concatenated features to the output labels. This network has five output nodes, one for each of the five possible labels (misogynous or not, and the four subcategories).
  • Sigmoid Activation: A Sigmoid activation function is used for the output layer. This is crucial for multi-label classification, as it allows each label's probability to be independent, unlike Softmax which forces probabilities to sum to one.
  • Thresholding: A threshold of 0.5 is applied to the output probabilities to determine the final binary prediction for each label.

The authors experimented with different architectures for the feed-forward network, including variations with dropout and batch normalization, and found that using ResNet-101 as the image encoder yielded superior results.

Datasets

The primary dataset used in this work is the Multimedia Automatic Misogyny Identification (MaMi) dataset, specifically created for the SemEval Task 5 challenge.

  • Training Data: 10,000 instances.
  • Test Data: 1,000 instances.

Each instance in the MaMi dataset is annotated with five binary labels: misogyny, shaming, stereotype, objectification, and violence. The paper notes that these subcategories are not mutually exclusive, meaning a single meme can fall into multiple categories.

Key Insights and Ideas

  • CLIP's Versatility: The paper demonstrates that CLIP, while originally for object detection, is highly adaptable for nuanced tasks like hate speech detection in multi-modal content. Its ability to jointly process images and text is a significant advantage.
  • Multi-label Classification is Key: Recognizing that misogynistic memes can exhibit multiple forms of harm (e.g., being both stereotypical and objectifying) is critical. The multi-label approach allows for a more comprehensive understanding of the meme's harmful nature.
  • Image Encoder Matters: The choice of image encoder within the CLIP framework can significantly impact performance. The study found that using a ResNet-101 encoder led to better results compared to ViT-based encoders, particularly for Task B (subcategory classification).
  • Generalization through Regularization: The authors observed that initial models with more complex feed-forward networks struggled with generalization. Incorporating dropout and batch normalization improved performance, highlighting the importance of regularization techniques for this task.

Key Results

The proposed CLIP-based multi-modal approach achieved strong performance in the SemEval-2022 Task 5 challenge.

  • Task A (Misogyny Detection): The best model achieved an F1-score of 0.694.
  • Task B (Misogyny Subcategory Identification): The best model achieved an F1-score of 0.681.

These scores significantly outperformed the baseline models provided by the organizers, which were based on VGG-16 for image features and USE for text features. The paper highlights that their best model utilized a pre-trained ResNet-101 as the image encoder and a feed-forward network with 200 hidden nodes.

Limitations and Future Work

  • Dataset Size: While the MaMi dataset is substantial, the authors acknowledge that larger datasets could further improve model performance and generalization.
  • Error Analysis: The paper mentions that the best model achieved good results, but a detailed error analysis to understand specific failure modes is not extensively discussed.
  • Future Directions: The authors suggest exploring different architectures for the feed-forward network and potentially incorporating more advanced multi-modal fusion techniques to further enhance performance. They also implicitly suggest that further research into understanding the nuances of visual-linguistic interactions in hateful content is warranted.

Summary updated

BibTeX
@inproceedings{behzadi2022mitra,
  author = {Mitra Behzadi and Ali Derakhshan and Ian Harris},
  title = {{Mitra Behzadi at SemEval-2022 Task 5 : Multimedia Automatic Misogyny Identification method based on CLIP}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure