TL;DR
This paper tackles the challenging task of automatically identifying misogynistic content in multimedia (memes), which often combines text and images. The authors explore various late-fusion techniques to combine predictions from separate text and image models, finding that a late fusion of probabilities using Auto-sklearn yields the best results, particularly for the more complex multi-label misogyny classification task.
What Problem Does This Paper Address?
Misogyny, a pervasive form of hate speech, is increasingly prevalent on social media. Identifying it is crucial for combating online discrimination. Memes, a popular form of online content, present a unique challenge because their offensive nature often arises from the interplay between text and images, making single-modality analysis insufficient. Existing research has explored text-based misogyny detection and, to a lesser extent, image-based detection, but a robust multimodal approach that effectively fuses information from both modalities, especially for nuanced tasks like multi-label misogyny classification, remains an active area of research. This paper addresses this gap by proposing and evaluating a multimodal late-fusion system for identifying misogynous memes.
Methods
The core of the proposed method is a late fusion strategy. This means that separate models are trained for text and image analysis, and their outputs are combined after individual training.
- Text Model: The authors experimented with several Transformer-based models available on Hugging Face, including BERTweet, all-distilroberta-v1, all-miniLM-L6-v2, twitter-roberta-base-offensive, and twitter-xlm-roberta-base.
- Image Model: Convolutional Neural Networks (CNNs) were used for image analysis. They explored different preprocessing techniques for images:
- No preprocessing.
- Blacking out text regions.
- Inpainting (removing) text regions.
- Late Fusion: The outputs from the text and image models (specifically, logits, probabilities, and embeddings) were combined using various fusion methods:
- Simple averaging or taking the maximum of predictions.
- Weighted averaging.
- Auto-sklearn: This automated machine learning tool was used to explore a wide range of models and preprocessing approaches to find the optimal pipeline configuration and ensemble for the multimodal late fusion step. This is a key aspect of their methodology, as it automates the search for the best combination.
Datasets
The study used a dataset of 10,000 memes with corresponding text transcriptions for the SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification (MAMI) competition. The data was split into training, validation, and test sets. For sub-task A (binary classification), a standard stratified split was used. For sub-task B (multi-label classification), a more sophisticated "iterative train test split" method was employed to ensure better representation of label combinations in the splits, given the unbalanced nature of the data.
Key Insights and Ideas
- Text is King (Mostly): The text modality consistently provided more informative signals for misogyny detection than the image modality across both sub-tasks.
- Image Preprocessing Nuances: For image analysis, the authors found that simple preprocessing techniques like blacking out or inpainting text did not significantly improve performance and, in some cases, might even hinder it. This suggests that the text embedded within images might be providing valuable context rather than just noise for CNNs.
- Late Fusion Pays Off: Combining predictions from separate text and image models (late fusion) generally outperformed single-modality models.
- Auto-sklearn's Power: Using Auto-sklearn to automate the search for the best late fusion configuration and ensemble proved highly effective, especially for the complex multi-label task.
- Logits vs. Probabilities: While logits contain more raw decision information, normalized probabilities from models often proved more useful as input for late fusion, particularly for the binary sub-task A. This is attributed to differences in logit distributions across models.
Key Results
- Sub-task A (Binary Misogyny Classification):
- Image Model: EfficientNetB0 achieved the highest F1 macro average score of 0.6825.
- Text Model: BERTweet-base performed best with an F1 macro average of 0.8320.
- Multimodal Late Fusion: The best score was achieved using Auto-sklearn with probabilities from both text and image models, resulting in an F1 macro average of 0.636.
- Sub-task B (Multi-label Misogyny Classification):
- Image Model: EfficientNetB0 again performed best with an F1 macro average of 0.3477.
- Text Model: BERTweet-base also led in text models with an F1 macro average of 0.5785.
- Multimodal Late Fusion: The Auto-sklearn approach using probabilities from text and image models achieved the highest score with an F1 macro average of 0.632. This is particularly noteworthy as it matches the performance of the binary task, suggesting the model effectively learned to handle the more complex multi-label scenario.
Limitations and Future Work
- Overfitting Concerns: The authors noted that the multimodal strategy's performance was not as strong as expected for sub-task A compared to some baselines. They hypothesize this could be due to overfitting, potentially exacerbated by differences in train/test set distributions. This is contrasted with sub-task B, where their method performed better, reinforcing the overfitting hypothesis for sub-task A.
- Text Dominance: The significant advantage of text over image information suggests further research into more sophisticated image feature extraction or fusion techniques might be needed.
- Pruning and Complexity: The consistent performance of text models across both sub-tasks suggests potential for exploring model pruning techniques to reduce complexity without sacrificing performance.
- Generalizability: The authors suggest their work provides a foundation for exploring late fusion methods in other multimodal problems and domains, and for comparing them against end-to-end deep classifiers. Future studies could investigate the complexity and pruning of the required models.
Summary updated