TL;DR
This paper presents UMUTeam's approach to identifying and categorizing misogynous content in memes for SemEval-2022 Task 5. They achieved competitive results by combining various linguistic, textual (BERT), and visual (BEiT) embeddings, exploring both knowledge integration and ensemble learning strategies.
What Problem Does This Paper Address?
The proliferation of online hate speech, particularly misogyny, is a significant societal concern. Memes, with their combination of images and text, are a common medium for spreading such harmful content. Identifying and categorizing misogynous memes is challenging due to the multi-modal nature of the content and the subtle ways misogyny can be expressed. SemEval-2022 Task 5 specifically aimed to address this by providing a dataset of memes and tasks for both binary (misogynous vs. not) and multi-label (shaming, stereotype, objectification, violence) classification.
Methods
UMUTeam's approach is multi-modal, leveraging a combination of different feature representations. They extract:
- Linguistic Features (LF): Using the UMUTextStats tool, they gather language-independent features related to phonetics, syntax, semantics, pragmatics, and figurative language. They specifically selected features robust to English.
- Textual Embeddings:
- Non-contextual: fastText for sentence (SE) and word (WE) embeddings.
- Contextual: BERT (BF) for sentence embeddings, obtained similarly to S-BERT.
- Visual Embeddings (BI): Using BEiT, a self-supervised model trained on ImageNet-21k, which represents images as sequences of fixed-size elements.
These features are then fed into various neural network architectures, including shallow MLPs, deep learning networks with different layer shapes, CNNs, and BiLSTMs. They explored two main strategies for combining these features:
- Knowledge Integration: Using the combined features as input to a single neural network.
- Ensemble Learning: Combining the predictions of models trained on individual feature sets using soft voting (mode) or averaging probabilities (mean).
For the multi-label task, they treated each misogynous trait as a separate binary classification problem.
Datasets
- MAMI Dataset: The primary dataset for SemEval-2022 Task 5, consisting of memes with overlaid text. It was used for training and validation. The paper notes an initial imbalance in the dataset and describes subsampling to address this.
- Custom Validation Split: UMUTeam created their own validation set from the MAMI training data for hyperparameter tuning and model evaluation.
Key Insights and Ideas
- Multi-modality is Crucial: The paper strongly suggests that combining textual and visual information is essential for effective misogyny detection in memes.
- Contextual Embeddings Outperform Non-Contextual: BERT embeddings (BF) generally performed better than fastText embeddings (SE, WE) for textual analysis.
- Visual Embeddings Show Promise: BEiT embeddings (BI) demonstrated strong performance, particularly for objectification and shaming traits, sometimes outperforming BERT alone.
- Feature Combination Benefits: Combining linguistic features with BERT and BEiT embeddings (LF, BF, BI) in a single neural network generally improved results over individual feature sets.
- Ensemble Learning Challenges: Ensemble learning strategies, especially averaging, did not consistently yield better results than knowledge integration, and in some cases, performed worse.
- Class Imbalance is a Major Hurdle: The authors repeatedly highlight class imbalance as a significant challenge, impacting performance, especially on the official test set.
Key Results
- Task A (Binary Classification): UMUTeam achieved position 36 out of 83 teams with a macro F1-score of 0.687 on the official leaderboard. Their best custom validation result for the combined LF, BF, BI (mode) feature set was a macro F1-score of 0.83913.
- Task B (Multi-label Classification): They secured position 28 out of 48 teams with a macro F1-score of 0.663 on the official leaderboard. Their best custom validation result for the LF, BF, BI feature set was a macro F1-score of 0.69457.
- Performance of LF: Linguistic features alone showed limited performance, especially due to some dictionaries not being available in English.
- BEiT vs. BERT: For Task A, BEiT (BI) achieved a macro F1-score of 0.83498, outperforming BERT (BF) which scored 0.81018.
- Violence Trait Anomaly: For the violence trait in Task B, one model achieved perfect recall, suggesting it might have been predicting all instances as violence, highlighting potential issues with imbalance or model overfitting.
Limitations and Future Work
- Class Imbalance: The authors acknowledge that handling class imbalance, particularly on the testing dataset, was a limitation. They suspect it contributed to performance discrepancies between validation and test sets.
- Limited English Linguistic Features: The availability of only a subset of language-independent linguistic features due to English translation issues limited their potential.
- Future Work: The authors suggest exploring data augmentation techniques for both images and text to better address class imbalance. They also plan to evaluate more sophisticated ensemble strategies and potentially other multi-modal fusion techniques.
Summary updated