TL;DR
This paper tackles the challenging task of automatically detecting and classifying misogynous memes by leveraging both visual and textual information. The authors propose novel strategies to mitigate biases in unimodal data and improve multimodal understanding, achieving competitive results in the SemEval-2022 MAMI competition.
What Problem Does This Paper Address?
Online misogyny, particularly in the form of gender-based violence and sexual harassment, is a pervasive issue. Memes, as a popular communication tool on social media, can be potent vehicles for spreading misogynistic content. Detecting these memes is complex because it requires understanding the interplay between visual elements and text, which can be subtle, context-dependent, and sometimes intentionally misleading. The SemEval-2022 Task 5 (MAMI) aimed to address this by asking participants to identify misogynous memes (Subtask A) and classify them into categories like stereotype, shaming, objectification, and violence (Subtask B). The challenge lies in models potentially overfitting to unimodal patterns (e.g., relying too heavily on text or image alone) and the inherent biases present in such data.
Methods
The DD-TIG team employed a multimodal approach, building upon state-of-the-art vision-language transformer models. Their core model was ERNIE-ViL-Large, chosen after evaluating several pre-trained models.
For Subtask A (Misogynous Meme Detection), they introduced several key strategies:
- Biased Words Masking: To combat unintended bias where models associate certain words (like "kitchen") with misogyny due to their frequent co-occurrence in biased texts, they proposed masking these "identity terms" with a probability during training. This acts as a form of data augmentation.
- Image Captioning: Recognizing that visual and textual information can be weakly aligned in memes, they used an image captioning model to generate additional textual descriptions of the visual content. This enriched input helps bridge the semantic gap.
- Template Meme Handling: Many memes are generated from templates. The authors identified that models can incorrectly associate misogynous scores with templates themselves, not the specific content. They addressed this by:
- Additional Template Memes: Collecting and adding false positive samples (memes incorrectly flagged as misogynous) generated from templates to the training set.
- Template Substitution: Using image retrieval to identify template-based memes and potentially generating new meme variations to improve robustness.
- Ensemble Learning: Combining predictions from multiple models trained with different strategies (e.g., varying biased word masking) to improve generalizability and robustness.
- Perspective API: Integrating an external knowledge source (Perspective API) to assess text toxicity and linearly combine its scores with model predictions.
For Subtask B (Misogynous Category Classification), they transformed the multi-label problem into a multi-class problem, treating each possible combination of labels as a distinct class. They also employed oversampling on positive samples to address the imbalance in category labels. Finally, they combined the results of Subtask A and Subtask B, ensuring that if a meme was classified as non-misogynous in Subtask A, its Subtask B prediction was discarded.
Datasets
The primary dataset used was the MAMI dataset provided for SemEval-2022 Task 5. This dataset contains meme images, their text transcriptions, and annotations for misogyny detection (Subtask A) and category classification (Subtask B). The authors also augmented their data by collecting additional memes from the website IMGflip for template meme handling.
Key Insights and Ideas
- Unintended bias is a significant hurdle: Models can easily learn spurious correlations from biased language, leading to misclassifications even when the content isn't truly misogynous. Strategies like biased word masking are crucial for mitigating this.
- Multimodal alignment is tricky: The semantic gap between image and text in memes is a real challenge. Image captioning can help bridge this gap by providing richer textual context for visual elements.
- Template memes introduce unique biases: Memes generated from templates can lead models to associate biases with the template itself rather than the specific content, requiring specialized handling.
- External knowledge can boost performance: Integrating tools like Perspective API can provide valuable contextual information that improves detection accuracy.
- Transforming multi-label to multi-class can be effective: For a small number of labels, this transformation can simplify the problem and potentially lead to better results, especially when combined with data balancing techniques.
Key Results
The DD-TIG team achieved strong performance, ranking 2nd for Subtask A and 5th for Subtask B in the SemEval-2022 MAMI competition.
- Their final system for Subtask A, incorporating biased word masking, image captioning, additional template data, ensemble learning, and Perspective API, achieved an Accuracy of 79.4% and an F1-score of 79.3%.
- For Subtask B, their approach using oversampling and problem transformation, combined with Subtask A results, yielded an F1-score of 72.8%.
- They found that while biased word masking and template meme handling significantly improved performance, image captioning alone had a less pronounced effect, sometimes even adding noise.
Limitations and Future Work
- False Positives in Subtask A: The authors acknowledge that a considerable number of non-misogynous samples were misclassified as misogynous (false positives). They suggest that enlarging the dataset might help address this.
- Weak Performance on "Shaming" in Subtask B: The model struggled with the "shaming" category, which they attribute to vague definitions, overlap with other categories, and situations where text isn't directly targeting depicted characters.
- Unimodal Imbalance: They note that in multimodal understanding tasks, unimodal information can sometimes be imbalanced, and text might need to be emphasized more in template-based memes.
- Future Work: The paper implies that further exploration into handling unimodal imbalances and refining strategies for specific categories like "shaming" would be beneficial. Enlarging the dataset is also a clear path forward.
Summary updated