TL;DR
This paper introduces two novel multimodal deep learning models for identifying misogyny in image-text posts, crucial for maintaining healthy online environments. Their approach combines text and image analysis, achieving competitive results in a SemEval task by employing a multi-stage training framework and a model ensemble with post-processing.
What Problem Does This Paper Address?
The internet, particularly image-based social media like Twitter and Instagram, has become a significant platform for women's voices. However, it also amplifies gender discrimination and aggressive content, perpetuating harmful stereotypes and inequality. Identifying and filtering such misogynistic content is essential for a healthy online social environment. This paper tackles the challenge of automatically detecting misogyny in multimedia posts, which often combine images and text, making it a complex task that requires understanding the interplay between visual and textual cues.
Methods
The authors propose two main multimodal model architectures:
- Transformer Single-flow Structure: This model processes image and text features together in a single stream of transformer modules. The idea is to map multi-modal features into a shared vector space. They leverage pre-trained VisualBERT weights and combine image features from ResNet-152, EfficientNet-b2, EfficientNet-b4, and EfficientNet-b7 with text features from BERT-base-uncased.
- Double-tower Structure: This architecture uses two parallel models, one for image and one for text, with MLPs at the output for joint training. This approach is less reliant on extensive pre-training and is more flexible, making it suitable for tasks with limited training data. They use ResNet-18 for image features and Bertweet for text.
To further improve performance, they employ:
- Model Ensemble: A weighted average of the predictions from the single-flow and double-tower models.
- Three-stage Training Framework:
- Multi-task Binary Classification: Initially trains to predict misogyny, stereotype, shaming, objectification, and violence.
- Single-task Binary Classification: Uses external data to train a binary classifier for misogyny, specifically to augment the training set for Sub-task A.
- Post-processing: Refines predictions by using the misogyny prediction to correct classifications for other categories, recognizing that certain misogynistic sub-types imply overall misogyny.
Datasets
- MAMI Dataset (SemEval-2022 Task 5): This is the primary dataset used for training and evaluation. It consists of image-text posts.
- Sub-task A: Focuses on binary misogyny identification.
- Sub-task B: Focuses on identifying specific types of misogyny (stereotype, shaming, objectification, violence).
- External Data: Used to augment the training set for Sub-task A, specifically from the "Memotion task" (Sharma et al., 2020).
Key Insights and Ideas
- Multimodal Fusion is Crucial: Misogyny often relies on the interplay between images and text. Simple text-only or image-only models would likely miss many instances.
- Double-tower Architecture's Adaptability: For tasks with limited training data, the double-tower architecture, which learns cross-modal features directly from the task data, can outperform single-flow models that heavily rely on extensive pre-training on different domains.
- Pre-training Domain Mismatch: Standard image-text pre-training (e.g., on ImageNet, COCO) might not be optimal for social media text-image data where text is often OCR'd from images and carries emotional context.
- Post-processing for Refinement: A simple post-processing step, leveraging the relationship between overall misogyny and its sub-types, can significantly improve performance, especially for categories like shaming or stereotype.
- Humor as a Misdirection: Some misogynistic content is disguised as humor, making it harder to detect and leading to misclassifications.
Key Results
The proposed methods achieved competitive results in the SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification.
- Sub-task A: Ranked 12th with a macro F1-score of 0.746.
- Sub-task B: Ranked 10th with a macro F1-score of 0.706.
- The model ensemble with post-processing achieved the best results, outperforming the organizers' baseline by significant margins. Specifically, the post-processing method achieved a macro F1 of 0.746 for Sub-task A and 0.708 for Sub-task B.
- The double-tower model generally performed better than the single-flow model, especially without extensive domain-specific pre-training.
Limitations and Future Work
- Limitations:
- The single-flow model's performance is heavily dependent on pre-training, and standard pre-training tasks might not be ideal for social media content.
- Some misogynistic samples are subtle and difficult to detect, even with multimodal approaches, especially when humor is involved.
- The paper notes that the effect of defining Sub-task B as a single-task binary classification was not significantly better than multi-task binary classification, suggesting potential complexities in that approach.
- Future Work:
- Explore more unlabeled sample data for pre-training the transformer single-flow method.
- Develop better multimodal methods specifically for Twitter scenarios.
- Investigate how to better handle subtle misogyny and humor-disguised misogyny.
Summary updated