LastResort at SemEval-2022 Task 5: Towards Misogyny Identification using Visual Linguistic Model Ensembles And Task-Specific Pretraining

Samyak Agrawal, Radhika MamidiNAACL 2022

TL;DR

This paper tackles the challenge of automatically identifying misogyny in internet memes, which often combine text and images to spread harmful content disguised as humor. The authors propose a multimodal approach using pre-trained visual-linguistic models, achieving competitive results by leveraging task-specific pretraining and ensembling.

What Problem Does This Paper Address?

Internet memes have become a powerful tool for communication, but they can also be used to spread hate speech, including misogyny. Identifying misogyny in memes is particularly challenging because it often relies on the interplay between text and images, where the visual element can reinforce or even subvert the textual message. This makes it a multimodal problem, requiring models that can understand both modalities simultaneously. The SemEval-2022 Task 5 specifically addresses this by asking participants to identify misogynous memes (binary classification) and categorize the type of misogyny present (multi-label classification).

Methods

The researchers employed a combination of established and novel techniques. Their core approach involves using Visual-Linguistic (VL) models like OSCAR and UNITER, which are pre-trained on large datasets to understand the relationship between images and text. They experimented with transfer learning, fine-tuning these pre-trained models on the misogyny detection dataset.

A key innovation is task-specific pretraining. This involves further pretraining the VL models on objectives specifically designed for tasks similar to misogyny detection, such as hateful memes detection. They also explored ensembling multiple models (trained on different visual features) to improve robustness. For comparison, they also trained text-only models (BERT and RoBERTa) to understand the contribution of visual cues.

Datasets

The primary dataset used is the SemEval-2022 Task 5 dataset, which contains 10,000 memes. This dataset is divided into training and development sets. For the misogyny identification task, half of the data is marked as positive (misogynous), and the other half as negative. The positive examples are further annotated with overlapping subcategories of misogyny: stereotype, shaming, objectification, and violence. The authors also mention using checkpoints from models pre-trained on the hateful memes dataset (Kiela et al., 2021) and the VisualGenome dataset for image feature extraction.

Key Insights and Ideas

  • Multimodality Matters, But Text is Powerful: While the task is inherently multimodal, the study found that text-only models (especially RoBERTa large) performed surprisingly well, sometimes even outperforming multimodal models on the binary misogyny detection task. This suggests that textual cues alone can carry significant weight in identifying misogyny.
  • Visuals Crucial for Nuance: However, for the more granular task of categorizing the type of misogyny (sub-task B), VL models showed a more significant advantage over text-only models. This highlights that visual elements are essential for understanding the subtle nuances and specific forms of misogyny.
  • Task-Specific Pretraining is Beneficial: Pretraining models on related tasks (like hateful memes detection) before fine-tuning on the target misogyny dataset yielded better results, indicating that knowledge transfer from similar domains is effective.
  • Ensembling Boosts Performance: Combining predictions from multiple models, particularly those trained on different visual features, led to improved performance, demonstrating the value of ensemble methods for robustness.
  • Hateful Memes vs. Misogyny: Interestingly, models fine-tuned on the hateful memes dataset performed unexpectedly worse on this misogyny task. This suggests that while related, these tasks might have subtle differences that require specific adaptation.

Key Results

The authors achieved competitive results, ranking 5th on sub-task A (misogynous meme identification) and 10th on sub-task B (categorizing misogyny types) at SemEval-2022.

  • Sub-task A: Their best performing model was an ensemble of OSCAR models, task-specifically pre-trained on the hateful memes dataset, achieving an F1-macro score of 68.6. Text-only RoBERTa large also performed strongly with an F1-macro of 68.3.
  • Sub-task B: The ensemble of OSCAR models with task-specific pretraining on their own dataset achieved the highest F1-macro score of 69.1. This was notably better than text-only models, which scored around 52.6 (RoBERTa large).

Limitations and Future Work

The authors acknowledge that while their models achieved good results, there is still room for improvement. They note that the performance difference between text-only and multimodal models in sub-task A suggests that misogyny detection might not be an "utterly multimodal problem."

Future work directions include:

  • Experimenting with more advanced VL models.
  • Developing better techniques to address class imbalance, particularly for sub-task B.
  • Exploring more sophisticated ensembling methods like Rank Averaging, Power Averaging, and Simplex Optimization.
  • Investigating why fine-tuning on hateful memes data didn't yield better results and how to better bridge the gap between related tasks.

Summary updated

BibTeX
@inproceedings{agrawal2022lastresort,
  author = {Samyak Agrawal and Radhika Mamidi},
  title = {{LastResort at SemEval-2022 Task 5: Towards Misogyny Identification using Visual Linguistic Model Ensembles And Task-Specific Pretraining}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure