TL;DR
This paper tackles the challenge of detecting misogynistic memes in low-resource Dravidian languages (Tamil and Malayalam) by proposing a novel multimodal approach. By fusing features from both text (using XLM-RoBERTa) and images (using Vision Transformer), their model achieves promising results, outperforming unimodal approaches and securing competitive rankings in a shared task.
What Problem Does This Paper Address?
Misogyny is a pervasive issue online, often manifesting in the form of text-and-image memes. Detecting these harmful memes is crucial for creating safer digital spaces, but it's particularly challenging in low-resource languages like Tamil and Malayalam due to a lack of specialized datasets and tools. Existing research has often focused on unimodal approaches (text-only or image-only) or high-resource languages, leaving a gap in effectively addressing the multimodal and low-resource nature of misogynistic memes in Dravidian languages. This paper aims to bridge that gap by developing an automated system for this specific context.
Methods
The core of the proposed method is a multimodal feature fusion strategy.
- Text Feature Extraction: They leverage the pre-trained XLM-RoBERTa (XLM-R) model, a powerful transformer-based language model capable of handling multiple languages, to extract textual features from meme transcriptions.
- Image Feature Extraction: For visual information, they employ the Vision Transformer (ViT), a state-of-the-art model for image analysis.
- Feature Fusion: The extracted text and image embeddings (each 768-dimensional) are concatenated. This "early-fusion" approach combines features before feeding them into a classifier, allowing the model to learn from both modalities simultaneously.
- Classification: A custom neural network classifier, named MultimodalClassifier, is then trained on these fused features. It consists of an input layer, a hidden layer with ReLU activation for non-linear interactions, and an output layer to predict the class (misogynistic or not).
- Training: The model is trained using the Adam optimizer with a learning rate of 0.001, CrossEntropyLoss, and batches of 32.
Datasets
The study utilizes a custom dataset specifically curated for misogyny detection in Tamil and Malayalam memes. This dataset contains images paired with transcriptions and binary labels indicating whether the meme is misogynistic or not. The dataset is divided into training, development, and test sets for each language. The paper notes that the Tamil training set has 851 non-misogynistic and 285 misogynistic samples, while the Malayalam training set has 381 non-misogynistic and 259 misogynistic samples. The test sets for both languages do not have provided labels.
Key Insights and Ideas
- Multimodality is Key: The most significant insight is that combining text and image features dramatically improves the accuracy of misogynistic meme detection compared to using either modality alone. This highlights how memes often rely on the interplay between visuals and text to convey their harmful messages.
- Low-Resource Languages are Underserved: The paper underscores the critical need for research and resources in low-resource languages, where the problem of online hate speech and misogyny is often overlooked but still significant.
- Pre-trained Models are Essential: Given the limited size of the custom dataset, the reliance on powerful pre-trained models like XLM-R and ViT is crucial for achieving good performance. This demonstrates the effectiveness of transfer learning in tackling complex tasks in data-scarce scenarios.
Key Results
The proposed multimodal model achieved promising results:
- Malayalam: A macro F1-score of 0.84066 on the test dataset, securing Rank 7 in the shared task.
- Tamil: A macro F1-score of 0.68830 on the test dataset, securing Rank 11 in the shared task.
These results are notable, especially considering the challenges of low-resource languages and the inherent class imbalance in misogynistic content. The paper also shows that their multimodal approach significantly outperforms unimodal models, as indicated by the performance metrics on the development datasets (Table 1).
Limitations and Future Work
- Limited Dataset Size: The authors acknowledge that the limited size of the labeled dataset is a constraint. This necessitated the use of pre-trained models, which might hinder the model's ability to capture highly nuanced, language-specific, or evolving forms of misogyny.
- Reliance on Pre-trained Models: While beneficial, the dependency on pre-trained models might affect the generalizability of the results to broader online contexts or entirely new linguistic variations.
- Future Directions: The authors suggest that future work could focus on acquiring larger, more culturally diverse, and real-world datasets to enhance the model's generalizability. They also emphasize the ongoing evolution of harmful discourse, highlighting the need for continuous advancements in multimodal AI models to create safer online environments.
Summary updated