IIITH at SemEval-2022 Task 5: A comparative study of deep learning models for identifying misogynous memes

Tathagata Raha, Sagar Joshi, Vasudeva VarmaNAACL 2022

TL;DR

This paper presents a comparative study of various deep learning models for identifying misogynous memes, a challenging task due to the multimodal nature of memes and the subtle ways misogyny can be expressed. The authors found that pretrained multimodal transformer architectures, especially those further fine-tuned on hateful meme datasets, significantly outperform simpler fusion methods and unimodal models, highlighting the power of task-adaptive pretraining.

What Problem Does This Paper Address?

Memes, while often used for humor, are increasingly becoming a vehicle for spreading hate speech, including misogyny and sexism. Identifying misogyny in memes is particularly challenging because it requires understanding both the visual and textual components, and misogyny can be expressed implicitly through stereotypes, shaming, objectification, or violence. The SemEval-2022 Task 5 aimed to address this by proposing two subtasks: a binary classification of whether a meme is misogynous, and a fine-grained classification of the specific type of misogyny present. This paper tackles the problem of building effective deep learning models for this multimodal misogyny identification task.

Methods

The authors explored a range of deep learning architectures, starting with unimodal models (using either text or image alone) as baselines. They then moved to multimodal approaches, which combine information from both modalities.

  • Unimodal Models: These included image-based classifiers using ResNet-152 features (Image-Grid) and region-based image features from Faster-RCNN (Image-Region), as well as text-based models using BERT and RoBERTa embeddings.
  • Unimodal Fusions: Simple fusion techniques were employed, such as concatenating unimodal representations (Concat-BERT) and averaging their outputs (Late Fusion).
  • Multimodal Transformers: The core of their investigation involved advanced transformer architectures designed for multimodal understanding. They experimented with MMBT, ViLBERT, and VisualBERT, utilizing versions pretrained on unimodal or multimodal objectives.
  • Task-Adaptive Further Pretraining: A key innovation was further pretraining the best-performing multimodal transformer models (VilBERT and VisualBERT) on a dataset specifically for classifying hateful memes. This "task-adaptive" pretraining aims to adapt the models to the nuances of meme-based hate speech.

Datasets

The primary dataset used for this study was provided by the SemEval-2022 Task 5 organizers.

  • Training Set: 10,000 memes.
  • Hidden Test Set: 1,000 memes. Each data point included a unique identifier, the path to the image, the meme's text, and labels for misogyny (binary) and specific misogyny types (stereotype, shaming, objectification, violence).

The authors also mention using other datasets for pretraining the multimodal transformer models:

  • Visual Genome: Used for pretraining image region features.
  • Conceptual Captions (CC): Used for multimodal pretraining of ViLBERT.
  • Common Objects in Context (COCO): Used for multimodal pretraining of VisualBERT.
  • Hateful Memes dataset: Used for task-adaptive further pretraining of VilBERT and VisualBERT.

Key Insights and Ideas

  • Multimodality is Crucial: Unimodal models, while serving as useful baselines, are significantly outperformed by multimodal approaches, underscoring the necessity of processing both image and text for accurate misogyny detection in memes.
  • Transformers Shine: Advanced multimodal transformer architectures like ViLBERT and VisualBERT demonstrate superior performance compared to simpler fusion methods, indicating their effectiveness in capturing complex cross-modal interactions.
  • Task-Adaptive Pretraining is Powerful: Further pretraining models on a related task (like hateful meme classification) provides a substantial performance boost. This suggests that adapting general multimodal models to the specific domain and nature of hateful memes is highly beneficial.
  • Subtle Misogyny is Hard: Even the best models achieved scores that indicate room for improvement, highlighting the inherent difficulty in detecting nuanced or implicit forms of misogyny.

Key Results

The paper reports macro F1 scores for Subtask 1 (course-grained identification) and weighted F1 scores for Subtask 2 (fine-grained identification).

  • Best Model Performance:
    • Subtask 1: VilBERT pretrained on the Hateful Memes dataset achieved a macro F1 score of 0.712.
  • Subtask 2: VisualBERT pretrained on the Hateful Memes dataset achieved a weighted F1 score of 0.706.
  • Comparison: The best multimodal transformer models (VilBERT and VisualBERT) significantly outperformed unimodal models and simpler fusion techniques. For instance, unimodal text models achieved around 0.62 F1, while fusion models reached about 0.61-0.64. The multimodal transformers pushed this higher, with VilBERT and VisualBERT achieving around 0.68-0.69 before further pretraining.

Limitations and Future Work

  • Hyperparameter Tuning: The authors acknowledge that they did not perform extensive hyperparameter tuning, suggesting that further optimization could lead to improved model performance.
  • Ensemble Methods: They suggest that exploring ensemble models, similar to Vilio (Muennighoff, 2020), could lead to even better results.
  • Challenging Nature of the Task: The scores, while improved, still indicate that detecting misogyny in memes remains a difficult problem, especially for subtle or implicit forms. This suggests a need for more sophisticated models or larger, more diverse datasets.

Summary updated

BibTeX
@inproceedings{raha2022iiith,
  author = {Tathagata Raha and Sagar Joshi and Vasudeva Varma},
  title = {{IIITH at SemEval-2022 Task 5: A comparative study of deep learning models for identifying misogynous memes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure