TL;DR
This paper presents a novel multi-modal approach for detecting misogynistic memes, combining image and text analysis using a graph network. The proposed system achieves competitive results on the SemEval-2022 Task 5, outperforming baselines by effectively leveraging pre-trained models and a graph-based classification module for multi-label misogyny type identification.
What Problem Does This Paper Address?
The internet has become a breeding ground for misogynistic content, often disguised as humor in the form of memes. These memes, while seemingly innocuous, contribute to online harassment and reinforce harmful gender stereotypes. Detecting and categorizing these misogynistic memes is crucial for mitigating their negative impact on individuals and society. The challenge lies in effectively combining visual and textual information within memes, as well as accurately identifying the specific types of misogyny present (e.g., shaming, objectification, stereotype).
Methods
The authors propose a multi-modal system that processes meme images and their accompanying text transcriptions. The core of their approach involves a Feature Extraction Module that handles each modality separately. For images, they utilize pre-trained CNN models like ResNet-101 and ResNet-50, incorporating a Navigator Module to identify informative regions within the image. Text is processed using the Sentence-BERT model to generate embeddings.
For the binary classification task (SubTask-A: misogynous or not), these extracted features are concatenated and fed into a linear layer for prediction.
For the multi-label classification task (SubTask-B: identifying the type of misogyny), they introduce a Graph Classification Module. This module treats each misogyny type as a node in a graph and uses a graph network to model the inter-dependencies between these labels. An adjacency matrix is constructed based on label co-occurrence patterns in the training data. This graph-based approach allows for a more nuanced understanding of how different types of misogyny relate to each other.
Datasets
The study utilizes datasets provided by the SemEval-2022 Task 5 organizers. These datasets consist of memes collected from the web and manually annotated for misogyny and its specific types. The paper notes a significant data imbalance across the different misogyny labels, which they address in their methodology.
Key Insights and Ideas
- Multi-modal fusion is key: Simply analyzing text or images in isolation is insufficient for accurate misogyny detection in memes. Combining both modalities significantly improves performance.
- Navigator Module for images: The navigator module, which helps identify relevant image regions, proved more effective than processing the entire image directly, suggesting that specific visual cues are critical for misogyny detection.
- Graph networks for multi-label relationships: Modeling the relationships between different types of misogyny using graph networks is crucial for accurately classifying the specific nature of the misogyny. This goes beyond simple independent classification of each type.
- Textual dominance in some cases: Ablation studies suggest that textual information might be more influential than visual information for certain types of misogynistic memes, possibly due to the meme's content or the quality of the image.
- Custom loss function for imbalance: A custom loss function that accounts for class imbalance and weighted importance of labels is effective in improving performance on the imbalanced dataset.
Key Results
The proposed Poirot system achieved competitive results in SemEval-2022 Task 5.
- For SubTask-A (binary classification), their model ranked 3rd out of 10 submissions, achieving a macro F1 score of 0.759 with the ResNet-50nsfw backbone.
- For SubTask-B (multi-label classification), they ranked 22nd out of 30 submissions, with their best model achieving a weighted F1 score of 0.645.
- Ablation studies showed that the navigator module improved image feature extraction and that textual features often played a more significant role than visual ones.
- The graph network depth was found to be sensitive, with deeper networks potentially leading to over-smoothing and performance degradation.
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Oversmoothing in Graph Networks: Increasing the depth of the graph convolution layers led to performance drops, likely due to over-smoothing. Future work aims to resolve this using effective normalization layers.
- Exploring alternative graph architectures: The authors plan to investigate other graph network approaches, such as Knowledge-Graph and GAT Networks, to better model multi-label dependencies.
- Image Quality: The paper suggests that the image content itself might not always be of high quality, which could impact the effectiveness of visual analysis.
- Domain-specific vs. General Representations: While fine-tuning on NSFW data showed promise, the paper also indicates that general feature representations might be more important for identifying the specificity of misogyny, suggesting a trade-off to explore.
Summary updated