TL;DR
This paper presents a deep learning system for automatically identifying misogynous content in online memes, combining text and image analysis. Their ensemble model, leveraging ALBERT for text and CNN for images, achieved the best performance by effectively fusing multimodal information.
What Problem Does This Paper Address?
The proliferation of social media has led to a surge in offensive content, particularly hate speech targeting women. Misogynistic memes, which combine images and text, are a significant challenge due to their often subtle and ironic nature. Automatically identifying and categorizing these memes is crucial for combating online harassment and promoting a safer digital environment. SemEval-2022 Task 5 specifically aimed to address this by focusing on "Multimedia Automatic Misogyny Identification" (MAMI), requiring systems to analyze both the visual and textual components of memes.
Methods
The TECHSSN team developed a multimodal deep learning architecture. For textual analysis, they explored transformer models, specifically BERT, ALBERT, and XLNET, leveraging their ability to understand contextual relationships in language. For image analysis, they employed Convolutional Neural Networks (CNNs), a standard for image feature extraction. The core of their approach lies in an ensemble prediction strategy. They trained separate transformer and CNN models and then combined their predictions using a majority voting mechanism. This ensemble approach aims to harness the strengths of both modalities, recognizing that misogyny can be conveyed through text, images, or a combination of both.
Datasets
The study utilized the dataset provided for SemEval-2022 Task 5. This dataset included training, trial, and testing splits with a total of 10,000 training samples. The data was categorized into misogynous and non-misogynous classes, with further sub-categorization of misogynous content into shaming, stereotype, objectification, and violence. The paper provides a detailed distribution of these categories within the training data, highlighting the imbalanced nature of some subtasks.
Key Insights and Ideas
- Multimodal Fusion is Key: The paper strongly suggests that analyzing memes requires considering both text and image. Relying on a single modality is insufficient for accurately detecting misogynistic content, as the interplay between text and visuals is often where the misogyny is embedded.
- ALBERT Outperforms Other Transformers for Text: For the textual component, ALBERT, a lighter variant of BERT, demonstrated superior performance compared to BERT and XLNET in this specific task. This indicates that for certain tasks, parameter efficiency can be achieved without significant performance degradation.
- Ensemble Power: The combination of text and image models through majority voting significantly improved performance over individual models, underscoring the benefit of a multimodal approach.
Key Results
The ensemble model, combining ALBERT for text and CNN for images, achieved the highest F1-scores for both subtasks.
- Subtask A (Misogynous vs. Non-Misogynous): The ensemble model achieved an F1-score of 0.5223. This was better than ALBERT alone (0.5128) and the CNN model alone (0.4756).
- Subtask B (Categorization of Misogynous Memes): The ensemble model also outperformed the BERT model alone, achieving an F1-score of 0.4673.
Limitations and Future Work
The authors acknowledge that training the text and visual modalities independently might lead to missed contextual information. They suggest that future work could explore more integrated approaches where text and visual information are processed jointly. This could involve developing more sophisticated fusion techniques or end-to-end multimodal transformer architectures. Additionally, they note that performance could be further improved by fine-tuning hyperparameters.
Summary updated