TL;DR
This paper tackles the challenge of identifying misogynistic memes, a form of hate speech that combines images and text. The authors propose a multimodal approach using the CLIP model, achieving a 71% F-score on the SemEval-2022 Task 5 MAMI challenge, demonstrating the effectiveness of combining visual and textual information for this complex task.
What Problem Does This Paper Address?
Hate speech is a growing problem on social media, and it's not just limited to text. Increasingly, hateful content appears in multimodal formats like memes, which combine images and text. Existing research on hate speech detection has largely focused on textual content, leaving the detection of multimodal hate speech, particularly misogyny in memes, under-addressed. This paper addresses this gap by focusing on the identification of misogynous memes, a task that requires understanding the interplay between visual and textual elements.
Methods
The authors employed a multimodal system based on the pre-trained CLIP (Contrastive Language–Image Pre-training) model. CLIP is known for its ability to learn joint representations of images and text, making it well-suited for this task. They extracted representations for both the image and the text transcription of each meme using CLIP. These representations were then combined in different ways to feed into a final classification layer:
- Text_Only: Used only the text embeddings from CLIP.
- Image_Only: Used only the image embeddings from CLIP.
- CLIP_concat: Concatenated the image and text embeddings into a single vector.
- CLIP_sum: Summed the image and text embeddings, with a trainable parameter to weigh their contributions. This "CLIP_sum" system achieved the best performance.
Datasets
The primary dataset used was provided for the SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification (MAMI) challenge. This dataset consists of 10,000 memes, with 5,000 labeled as misogynous and 5,000 as not misogynous. Each meme includes a JPG image and a text transcription of any overlay text. The competition also provided an additional set of 1,000 unlabeled memes for system evaluation.
Key Insights and Ideas
- Multimodality is Crucial: The results clearly show that combining image and text information (CLIP_concat and CLIP_sum) significantly outperforms using either modality alone (Text_Only and Image_Only). This underscores the importance of a multimodal approach for understanding nuanced hate speech in memes.
- CLIP's Effectiveness: The CLIP model, designed for joint image-text understanding, proves to be a powerful foundation for this task, enabling effective representation learning for both modalities.
- Text Dominance (to an extent): The "Text_Only" system achieved a surprisingly high F-score (69.23%), suggesting that in many cases, the text transcription alone provides sufficient clues for identifying misogyny in memes. This is attributed to the dataset's characteristic where text often carries the brunt of the hateful message.
- Potential for Bias: The error analysis revealed a tendency for the model to misclassify memes featuring male figures or inanimate objects as non-misogynistic (false negatives) and memes with female figures as misogynistic (false positives). This suggests a potential bias in the training data, where misogyny might be disproportionately associated with images of women.
Key Results
The authors reported a best F-score of 71% using their "CLIP_sum" system, which combined image and text representations with a learned weighting. Other systems performed as follows:
- Text_Only: 69.23% F-score
- Image_Only: 65.37% F-score
- CLIP_concat: 70.50% F-score
Limitations and Future Work
The authors acknowledge that the dataset might have a bias where misogynous memes are often paired with images of women. This could lead to the model learning spurious correlations. They suggest that a deeper analysis is needed to understand and mitigate this bias. Future work could also explore other multimodal architectures and investigate the detection of hate speech in other multimedia formats like videos, which are increasingly popular on social networks.
Summary updated