TL;DR
This paper presents a multi-modal system for detecting misogynous memes by fusing image and text features using multi-task learning. The system achieved top performance on the SemEval-2022 Task 5, demonstrating the effectiveness of combining advanced deep learning models and a novel approach to correcting misogyny predictions based on sub-categories.
What Problem Does This Paper Address?
The internet, particularly social media, has become a breeding ground for hate speech, including misogyny. Memes, a popular form of online communication, are increasingly used to spread sexist and aggressive messages, exacerbating gender inequality. Manually moderating this vast amount of content is impossible, necessitating automated detection systems. While previous work has focused on text-based misogyny detection, memes inherently combine visual and textual elements, making multi-modal analysis crucial for accurate identification. This paper tackles the challenge of effectively integrating image and text information to detect misogynous memes, a task that is complex due to the nuanced and often overlapping nature of misogynistic content.
Methods
The authors propose a multi-modal system that leverages multi-task learning and a multi-model fusion approach.
- Image Feature Extraction: They employ a fine-tuned ConvNeXt model, a powerful convolutional neural network architecture, to extract image features.
- Text Feature Extraction: Two state-of-the-art pre-trained language models, RoBERTa and DeBERTa, are used to extract text features.
- Modality Fusion: Image and text features are fused by direct splicing, feeding into a classification layer. This approach is described as "early fusion" in the context of combining features before final prediction.
- Multi-Task Learning: The system is designed to handle two subtasks: Task A (binary classification: misogynous or not) and Task B (multi-label classification of specific misogyny types: stereotype, shaming, objectification, violence).
- Model Fusion: The final predictions are generated by a weighted average of results from two fusion models (ConvNeXt + RoBERTa and ConvNeXt + DeBERTa).
- Post-processing Correction: A key innovation is the correction of Task A (misogynous detection) results by leveraging the predictions from Task B. Specifically, if any of the sub-categories (shaming, stereotype, objectification, violence) are predicted as positive, the "misogynous" label is also confirmed. This post-hoc correction significantly improved Task A performance.
Datasets
The study uses datasets provided for the SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification (MAMI) competition.
- Training Set: 10,000 memes (images) with corresponding CSV files for annotations.
- Testing Set: 1,000 memes (images) with corresponding CSV files for annotations. The dataset includes labels for "misogynous" (binary) and four sub-categories: "stereotype," "shaming," "objectification," and "violence" (all binary). The paper notes that while the "misogynous" label is balanced, the sub-categories are unbalanced.
Key Insights and Ideas
- Multi-modal is Crucial: The paper strongly supports the idea that combining image and text information is superior to using either modality alone for misogynous meme detection.
- ConvNeXt's Strength: The ConvNeXt architecture proved to be a more effective image feature extractor than Swin Transformers in their system.
- Simple Fusion Works Best: Direct concatenation of image and text features (early fusion) yielded better results than more complex fusion methods like LFN.
- Post-hoc Correction is Powerful: Correcting the binary misogynous prediction (Task A) based on the multi-label predictions (Task B) provided a significant performance boost, highlighting the importance of understanding the nuances of misogyny.
- Multi-task Learning Benefits: Training the system to perform both binary and multi-label classification simultaneously improved overall performance compared to individual training.
Key Results
The proposed system achieved excellent results in the SemEval-2022 Task 5:
- Task A (Misogynous Detection): 0.755 accuracy (ranking 3rd on the leaderboard).
- Task B (Multi-label Classification): 0.731 accuracy (ranking 1st on the leaderboard).
The paper also reports that the weighted average fusion of ConvNeXt + DeBERTa and ConvNeXt + RoBERTa, with a threshold of 0.405, yielded the best results.
Limitations and Future Work
- Dataset Imbalance: The authors acknowledge the imbalance in the sub-category labels within the dataset, which can pose challenges for training.
- Error in Training Data: They note some errors in the training data where memes labeled "No Misogynous" still contained positive labels for sub-categories.
- Threshold Tuning: The paper explores various threshold values for fusion, indicating that careful tuning is critical for optimal performance.
- Future Work: While not explicitly detailed, the success of their multi-modal, multi-task, and post-processing correction approach suggests that further exploration of these techniques, potentially with even more advanced models or fusion strategies, could lead to further improvements. The paper also hints at the potential for using sub-category results to correct misogynous predictions, which they found effective.
Summary updated