One_by_zero@DravidianLangTech 2025: A Multimodal Approach for Misogyny Meme Detection in Malayalam Leveraging Visual and Textual Features

Dola Chakraborty, Shamima Afroz, Jawad Hossain, Mohammed Moshiul HoqueNAACL 2025

TL;DR

This paper tackles the challenging task of detecting misogynistic memes in Malayalam, a low-resource language, by employing a multimodal approach that combines visual and textual features. Their system, leveraging ResNet18 for images and IndicBERT for text, achieved a strong F1-score of 0.87, securing 3rd place in the DravidianLangTech@NAACL 2025 shared task. This work highlights the effectiveness of multimodal analysis for nuanced online content detection, especially in under-resourced linguistic contexts.

What Problem Does This Paper Address?

Misogynistic memes are a pervasive online issue, spreading harmful ideas about women through a combination of images and text. While detecting such content in high-resource languages like English has seen significant research, low-resource languages, particularly Dravidian languages like Malayalam, remain largely underexplored. This gap is critical because misogynistic memes can subtly embed and normalize sexist beliefs, making them difficult to identify and combat. The challenge is amplified by the multimodal nature of memes, where the offensive message might only emerge when text and image are considered together, and by linguistic nuances, transliterated text, and regional cultural references specific to Malayalam. The lack of annotated datasets further exacerbates this problem.

Methods

The authors propose a multimodal approach for misogyny meme detection. For visual feature extraction, they experimented with several pre-trained models including Vision Transformer (ViT), VGG16, and ResNet-18, ultimately selecting ResNet-18 for their best-performing model. For textual feature extraction, they utilized advanced language models like Malayalam BERT and IndicBERT, with IndicBERT being chosen for the final multimodal model.

The core of their approach lies in multimodal fusion. They concatenate the extracted visual and textual embeddings and pass them through fully connected layers to reduce dimensionality and align features. This combined representation is then fed into a final classification layer with a sigmoid activation to predict whether a meme is misogynistic or not. The entire system was trained end-to-end using a binary cross-entropy loss function and the AdamW optimizer.

Datasets

The paper utilizes a dataset specifically created for the DravidianLangTech@NAACL 2025 shared task on Misogyny Meme Detection in Malayalam. This dataset consists of memes with associated transcriptions. The authors provide a breakdown of the dataset's distribution:

  • Training Set: 640 samples (259 misogynistic, 381 non-misogynistic)
  • Validation Set: 160 samples (63 misogynistic, 97 non-misogynistic)
  • Test Set: 200 samples (78 misogynistic, 122 non-misogynistic)

Key Insights and Ideas

  • Multimodal fusion is crucial: The paper strongly demonstrates that combining visual and textual information significantly outperforms unimodal approaches for misogyny meme detection, especially in complex cases where the offense is implicit.
  • IndicBERT's effectiveness in LRLs: The success of IndicBERT, a multilingual model, over a language-specific model like MalayalamBERT for textual analysis highlights its potential for cross-lingual effectiveness in Dravidian languages. This is a significant finding for low-resource language NLP.
  • Implicit misogyny is a major challenge: The error analysis reveals that the model struggles with memes where the textual content appears non-misogynistic but carries misogynistic undertones when paired with the image, or when dealing with sarcasm, slang, and culturally specific humor.

Key Results

The authors' multimodal approach, specifically the combination of ResNet-18 (visual) and IndicBERT (textual), achieved the highest performance on the test set, yielding a macro F1-score of 0.87. This result earned them the 3rd rank in the shared task.

  • Visual Approach: Vision Transformer (ViT) achieved the highest F1-score of 0.79.
  • Textual Approach: Malayalam BERT achieved the highest F1-score of 0.86.
  • Multimodal Approach: The fusion of ResNet-18 and Malayalam BERT achieved an F1-score of 0.86, while ResNet-18 and IndicBERT achieved the top F1-score of 0.87. The confusion matrix for the best model shows that out of 200 test samples, 176 were correctly predicted. The model misclassified 11 misogynistic samples as non-misogynistic and 13 non-misogynistic samples as misogynistic.

Limitations and Future Work

The authors acknowledge several limitations:

  • Confusion from feature combination: Combining features from ResNet-18 and IndicBERT can sometimes lead to confusion and misclassifications, particularly when dealing with implicit misogyny.
  • Dataset size: The relatively small dataset size may hinder the model's ability to capture nuanced patterns and generalize effectively.

Future work suggestions include:

  • Larger datasets: Incorporating larger datasets to improve model robustness, reduce bias, and enhance generalization.
  • Exploring other models: Investigating larger transformer-based models for multimodal learning and exploring large language models.
  • Cross-lingual effectiveness: Further exploring IndicBERT's potential for cross-lingual effectiveness in other Dravidian languages.

Summary updated

BibTeX
@inproceedings{chakraborty2025one,
  author = {Dola Chakraborty and Shamima Afroz and Jawad Hossain and Mohammed Moshiul Hoque},
  title = {{One\_by\_zero@DravidianLangTech 2025: A Multimodal Approach for Misogyny Meme Detection in Malayalam Leveraging Visual and Textual Features}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure