TL;DR
This paper introduces a multimodal deep learning framework using transformers to detect misogynistic memes in low-resource Dravidian languages (Tamil and Malayalam). By combining text and image features, the model achieves promising F1-scores, addressing the challenge of subtle misogyny and code-mixing in these languages.
What Problem Does This Paper Address?
The internet is awash with memes, which, while often humorous, can also be a vehicle for harmful content like misogyny. Detecting misogynistic memes is crucial for fostering a more respectful online environment. However, this task is particularly challenging in low-resource languages like Tamil and Malayalam due to:
- Linguistic Complexity: These languages have unique syntactic structures, vocabulary, and idiomatic expressions, making it difficult for standard NLP models.
- Data Scarcity: Labeled datasets for misogyny detection in these languages are scarce, hindering the training of effective machine learning models.
- Multimodality: Memes combine images and text, where the visual elements often carry crucial context that text-only models miss. This interplay between text and image makes detection complex.
- Code-Mixing: Memes often feature a mix of languages, further complicating analysis.
This paper aims to tackle these challenges by developing a multimodal approach specifically for Tamil and Malayalam memes.
Methods
The proposed framework employs a transformer-based multimodal approach for misogyny meme detection. The key steps are:
- Preprocessing:
- Text: Noise like stopwords, digits, and punctuation are removed from meme transcriptions.
- Images: Images are resized to a fixed input size (224x224 pixels) for the Vision Transformer.
- Feature Extraction:
- Text: XLM-ROBERTa, a multilingual transformer model, is used to extract text features. It tokenizes text using Byte-Pair Encoding, handles rare words, and generates high-dimensional embeddings that capture token, positional, and segment information.
- Images: Vision Transformer (ViT) is used for image feature extraction. Images are split into patches, flattened, and projected into a high-dimensional space. These patch embeddings, along with positional encodings, are processed through self-attention layers to capture local and global contextual features.
- Feature Fusion: A simple concatenation strategy is used to combine the extracted text and image features.
- Classification:
- Machine Learning Models: K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Random Forest (RF), and Naive Bayes (NB) are trained on the fused features.
- Deep Learning Models: Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks are employed to capture temporal dependencies.
- Multimodal Classifier (MMC): A custom two-stage deep learning model is proposed. It uses separate subnetworks for text and image features, followed by a three-layer fusion network to learn inter-modal relationships before final classification using a softmax activation. This MMC is trained end-to-end with Adam optimizer.
Datasets
The study utilizes two datasets for Tamil and Malayalam:
- Tamil Dataset: 1,133 memes for training and 356 for testing.
- Malayalam Dataset: 640 memes for training and 200 for testing.
Both datasets contain pictorial content and overlaid text. A key characteristic noted is that the datasets are imbalanced, with a significantly higher number of non-misogynistic memes compared to misogynistic ones.
Key Insights and Ideas
- Multimodality is Key: The paper strongly advocates for multimodal approaches, highlighting that visual elements in memes are crucial for conveying context and meaning that text alone cannot capture.
- Transformer Power: Leveraging pre-trained transformer models like XLM-ROBERTa and Vision Transformer proves effective for extracting rich features from both text and images, even in low-resource languages.
- Addressing Low-Resource Challenges: The framework is designed to handle the complexities of low-resource languages, including code-mixing and data scarcity, by using robust feature extraction and fusion techniques.
- Fusion Matters: Simple concatenation of features from text and image modalities is shown to be effective, allowing the model to leverage both types of information for classification.
Key Results
The paper reports the performance of various classifiers on the fused embeddings:
- Tamil Dataset: The Multimodal Classifier (MMC) achieved the best performance with an F₁-score of 0.77.
- Malayalam Dataset: The GRU model achieved the highest performance with an F₁-score of 0.88. The proposed MMC model also performed comparably with an F₁-score of 0.86.
These results demonstrate the model's success in detecting misogynistic content, even with code-mixing and limited training data.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Size: The current dataset size might not be sufficient to generalize across all subtle forms of misogyny, especially those involving implicit biases or sarcasm.
- Complex Content: The model struggles with code-mixed content, low-resolution images, and heavily stylized fonts, which can obscure textual and visual understanding.
- Language Discrepancies: The better performance on Malayalam compared to Tamil might stem from dataset composition, linguistic variations, or image-text alignment differences.
For future work, the authors suggest developing a stronger system by addressing these limitations, potentially through more advanced data augmentation, improved handling of code-mixing, and larger, more diverse datasets.
Summary updated