TL;DR
This paper tackles the challenging task of identifying misogyny in internet memes by combining text and image analysis. The authors explored various deep learning architectures, finding that a BiLSTM model incorporating both BERT embeddings for text and a nudity detection module for images achieved the best performance, highlighting the importance of multimodal analysis for this complex problem.
What Problem Does This Paper Address?
Misogyny, a pervasive form of sexism, has found a new and insidious outlet in internet memes. While text-based hate speech detection is a well-studied area, memes present a unique challenge due to their multimodal nature, often relying on the interplay between images and text, frequently employing humor and sarcasm to mask their harmful intent. This makes traditional text-only analysis insufficient. The SemEval-2022 Task 5 aimed to address this by requiring participants to develop models capable of binary classification of memes as misogynistic or not, pushing the boundaries of multimodal sentiment analysis and online abuse detection.
Methods
The researchers explored several deep learning architectures for meme misogyny identification. Their approach involved leveraging powerful pre-trained language models like BERT and RoBERTa for text encoding, recognizing their ability to capture contextual nuances in word embeddings, a significant improvement over static embeddings like Word2Vec. For image analysis, they employed the NudeNet tool, a nudity classification algorithm, to detect potentially "unsafe" content.
Several models were developed and compared:
- BiLSTM Neural Network with RoBERTa/BERT embeddings: These models focused on text processing using recurrent neural networks with advanced transformer-based embeddings.
- BiLSTM Neural Network with BERT as embedding + nude detection: This approach combined text analysis with image-based nudity detection, exploring how this multimodal fusion impacts performance. Different thresholds for the nudity detection score were investigated.
- 1-Dimensional Convolutional Neural Network with BERT as embedding + nude detection: This model explored the use of CNNs for text processing, again combined with image analysis.
- BERT Transformer with nude detection: This model directly utilized BERT's transformer architecture for text and integrated nudity detection.
A key aspect of their methodology was the preprocessing of text data, which included lowercasing, removing URLs, hashtags, mentions, emojis, punctuation, numbers, and short words, as well as filtering English stop words.
Datasets
The study utilized a dataset specifically curated for SemEval-2022 Task 5, Subtask A. This dataset comprised 5000 memes labeled as misogynistic (value 1) and 5000 labeled as not misogynistic (value 0), making it a well-balanced dataset. Each entry included the image file name and the extracted text transcription. An additional "unsafe" column was added for models that incorporated nudity detection.
Key Insights and Ideas
- Multimodality is Crucial: The most significant insight is that combining text and image analysis demonstrably improves misogyny detection in memes compared to using either modality alone. This underscores the inherent nature of memes as a multimodal communication form.
- Nudity Detection as a Proxy: The use of a nudity detection classifier (NudeNet) proved surprisingly effective. The authors found that images flagged as "unsafe" by NudeNet often correlated with misogynistic content, suggesting that certain visual cues, even if not explicitly violent, can be indicative of harmful intent in the context of memes.
- BERT's Contextual Power: The superiority of BERT and RoBERTa embeddings over traditional methods like Word2Vec highlights the importance of contextual understanding in natural language processing for tasks involving nuanced language like sarcasm and irony, which are common in memes.
- Fusion Strategies Matter: The specific way text and image information are fused can significantly impact performance. The best-performing model integrated nudity detection with a carefully chosen threshold, suggesting that not all "unsafe" images contribute equally to misogyny.
Key Results
The paper reports that the best-performing model, a BiLSTM Neural Network with BERT as embedding and nude detection (using a threshold of 0.45), achieved an F1-score of 0.665 in the evaluation phase. This result placed them 43rd in the overall ranking for the task. While the F1-scores for most models hovered around 0.60-0.65, the multimodal approach consistently outperformed single-modality models.
Limitations and Future Work
The authors acknowledge that their image classifiers could be further improved. They also express an interest in exploring other pre-trained transformer models for future work. The reliance on a nudity detection tool, while effective, might not capture all forms of visual misogyny that don't involve nudity. Future research could focus on developing more sophisticated image analysis techniques tailored to the specific visual tropes of misogynistic memes.
Summary updated