TL;DR
This paper tackles the challenging task of automatically identifying and classifying misogynous memes, which combine text and images to convey hateful messages. The researchers developed an ensemble learning approach, combining multiple text and image models, to achieve state-of-the-art performance on the SemEval-2022 Task 5 MAMI dataset.
What Problem Does This Paper Address?
Misogyny is a pervasive issue, and its presence in online content, particularly memes, poses a significant challenge for automated detection. Memes, by their nature, often combine text and images in creative and sometimes subtle ways, making them difficult for traditional content moderation systems to analyze. The SemEval-2022 Task 5: MAMI (Multimedia Automatic Misogyny Identification) aimed to address this by providing a dataset of English memes labeled for misogyny. The task was split into two parts: identifying if a meme is misogynous (binary classification) and classifying misogynous memes into specific sub-categories like shaming, stereotyping, objectification, and violence (multi-label classification). The inherent multi-modal nature of memes, where the meaning can arise from the interplay between text and image, makes this a complex problem.
Methods
The core of this work lies in an ensemble learning strategy that combines multiple uni-modal (text-only or image-only) and bi-modal (text and image) models.
Uni-modal Models: * BERTC: A pre-trained RoBERTa model fine-tuned for text classification. * GCAN: A Graph Convolutional Attention Network that leverages RoBERTa word embeddings. This model treats words as nodes in a graph and uses co-occurrence and TF-IDF to define edge weights, aiming to capture richer contextual information than standard Transformers. * ViT: A pre-trained Vision Transformer model used for image classification.
Bi-modal Models: These models fuse the outputs of the uni-modal models using a proposed "model fusion network." This network concatenates classification probabilities and features from the uni-modal models. Two fusion strategies were explored: stream-weighting-based decision fusion and representation fusion. Several bi-modal combinations were created, such as BERTC-ViT, GCAN-ViT, BERTC-GCAN, and BERTC-GCAN-ViT.
Ensemble Learning: The researchers employed two levels of ensemble:
- Dataset-level Ensemble: A soft voting ensemble where multiple models of the same architecture (trained with different random seeds or folds) are combined.
- Model-level Ensemble: A hard voting strategy that combines the predictions of the best performing uni-modal and bi-modal models.
Datasets
The primary dataset used is the SemEval-2022 Task 5: MAMI dataset. This dataset contains 10,000 memes for training and 1,000 for testing, with manual annotations for misogyny and its sub-categories.
Key Insights and Ideas
- The Power of Textual Context: Text-only models (BERTC and GCAN) generally outperformed the image-only ViT model, highlighting the significant role of textual content in identifying misogyny, even in multi-modal memes.
- Graph Convolutional Networks for Text: The GCAN model, by incorporating graph structures and word co-occurrence, demonstrated superior performance over a standard Transformer (RoBERTa), suggesting that capturing explicit relational information between words can be beneficial, especially for short texts found in memes.
- Bi-modal Fusion is Crucial: Combining text and image information through fusion networks significantly improved performance compared to uni-modal models. This underscores the importance of considering both modalities for accurate misogyny detection in memes.
- Ensembling for Robustness: Both dataset-level and model-level ensembling techniques were effective in further boosting performance, demonstrating that aggregating predictions from multiple diverse models leads to more robust and accurate results.
Key Results
The paper reports the following key results:
- Sub-task A (Binary Classification): The ensemble system achieved a macro-average F1-score of 0.755, ranking 11th in the competition. The GCAN-ViT bi-modal model individually performed very well.
- Sub-task B (Multi-label Classification): The ensemble system achieved a weighted-average F1-score of 0.709, ranking 10th.
- The best performing individual bi-modal model was GCAN-ViT.
- The "oracle results" (achieved after the challenge by combining the best possible subsets of models) showed even higher scores, with a combination of ViT, BERTC-GCAN-ViT, BERTC-ViT, and GCAN-ViT achieving 0.762 for sub-task A.
Limitations and Future Work
- OCR Errors: The paper acknowledges that OCR-based text transcriptions can be error-prone, which can negatively impact text-based models.
- Image Recognition Challenges: Images in memes can be difficult for automatic systems to recognize due to overlaid text, complex compositions, or artistic styles.
- Future Work: The authors suggest that further improvements could be made by exploring more sophisticated fusion techniques and potentially incorporating external knowledge bases or more advanced multi-modal Transformer architectures. The success of the GCAN model also points towards further research into graph-based methods for text representation in this domain.
Summary updated