TL;DR
This paper tackles the challenge of automatically detecting misogynistic memes, a growing problem on social media. The authors developed a multimodal deep learning system that combines visual and textual information from memes, achieving a competitive F1-score of 0.722 and ranking 17th in the SemEval-2022 Task 5 competition.
What Problem Does This Paper Address?
Online social networks have become a primary communication channel, but they also host harmful content, including misogynistic memes. These memes, often a blend of images and text, are designed to spread discrimination and hate against women. Detecting them is challenging because they leverage visual humor and cultural context, making simple text-based analysis insufficient. The SemEval-2022 Task 5 aimed to address this by identifying misogynous memes and categorizing the type of misogyny (stereotype, shaming, objectification, violence).
Methods
The researchers employed an ensemble approach, combining the predictions of multiple pre-trained multimodal deep learning models. Their core strategy involved using models that can process both images and text simultaneously. Specifically, they experimented with:
- Text-based models: RoBERTa, a powerful language model, was used as a baseline but found insufficient on its own.
- Image-based models: VGG-16, a well-known Convolutional Neural Network (CNN), was also tested but proved inadequate for meme analysis without text.
- Multimodal models: This is where the main innovation lies. They utilized:
- MMBT (Multimodal Bi-Transformers): This model uses transformer architectures to fuse image and text embeddings. They employed ResNet-152 for image feature extraction. Two MMBT variants were trained, differing in their BERT text encoder (base vs. large) and random seeds.
- VisualBERT: This model also leverages transformer layers for vision-language tasks. It uses a custom object detector (Detectron2 with MaskRCNN+ResNet-101+FPN) to extract visual embeddings.
The final system was an ensemble of the two MMBT models and VisualBERT, where their predictions were combined using a voting technique. This ensemble approach aims to leverage the strengths of each model to improve overall performance.
Datasets
The study used the official SemEval-2022 Task 5 dataset, which comprised:
- Training dataset: 10,000 memes with extracted text and labels for misogyny detection.
- Evaluation dataset: 1,000 memes for submission and evaluation.
The dataset was gathered from social media platforms like Twitter and Reddit, as well as meme-sharing websites. The training data was balanced, with 5,000 misogynous and 5,000 non-misogynous memes for sub-task A.
Key Insights and Ideas
- Multimodality is Crucial: The paper strongly reinforces the idea that memes cannot be effectively analyzed using text or images alone. Combining both modalities is essential for robust detection.
- Ensembling for Robustness: Combining multiple well-performing multimodal models through an ensemble technique led to a significant performance boost, outperforming individual models.
- MMBT's Strength: The MMBT models, particularly when ensembled, showed strong performance, indicating the effectiveness of their transformer-based fusion of visual and textual information.
- VisualBERT's Contribution: While MMBT slightly outperformed VisualBERT individually, VisualBERT still contributed positively to the final ensemble, highlighting the value of diverse model architectures.
Key Results
The proposed ensemble system achieved an F1-score of 0.722 on sub-task A (detecting misogynous memes). This performance placed them 17th out of 83 participating teams. This represents a notable improvement over the baseline model's F1-score of 0.65. The confusion matrix shows that the system correctly identified 359 non-misogynous and 363 misogynous memes, while misclassifying 137 misogynous memes as non-misogynous and 141 non-misogynous memes as misogynous.
Limitations and Future Work
- Limitations: The paper notes that the text-based RoBERTa and image-based VGG-16 models performed poorly, underscoring the necessity of multimodal approaches. While the ensemble improved performance, there's still room for error in misclassification.
- Future Work: The authors implicitly suggest that further exploration of different multimodal architectures and ensemble strategies could lead to even better results. The task of categorizing misogyny types (sub-task B) is also a complex area for future research.
Summary updated