Fired_from_NLP@DravidianLangTech 2025: A Multimodal Approach for Detecting Misogynistic Content in Tamil and Malayalam Memes

Md. Sajid Alam Chowdhury, Mostak Mahmud Chowdhury, Anik Mahmud Shanto, Jidan Al Abrar, Hasan MuradNAACL 2025

TL;DR

This paper tackles the crucial problem of detecting misogynistic content in memes from low-resource languages like Tamil and Malayalam, an area previously under-researched. The authors propose a multimodal approach combining visual and textual features, achieving impressive F1 scores of 0.678 for Tamil and 0.803 for Malayalam, highlighting the effectiveness of language-specific models.

What Problem Does This Paper Address?

The proliferation of social media has led to the widespread sharing of memes, which, while often humorous, can also be used to spread harmful ideologies like misogyny. Detecting such content is vital for maintaining a safe online environment. While research on harmful meme detection exists, most of it focuses on high-resource languages like English. There's a significant gap in research for low-resource languages such as Tamil and Malayalam, making it harder to combat misogyny in these diverse online communities. This paper aims to bridge this gap by developing a robust multimodal model for detecting misogynistic memes in these languages.

Methods

The authors propose a multimodal approach that leverages both visual and textual information from memes. The core of their methodology involves:

  • Input Modalities: They use both the text transcriptions of memes and the meme images themselves.
  • Preprocessing: Text is tokenized and padded/truncated to uniform length, with attention masking applied. Images are resized and normalized.
  • Feature Extraction:
    • Text: They experiment with several transformer-based models, including general-purpose multilingual models like mBERT and Indic-BERT, as well as language-specific models: Tamil-BERT and Malayalam-BERT. The pooler_output representation from the final transformer layer is extracted.
  • Image: They utilize popular Convolutional Neural Network (CNN) architectures pre-trained on ImageNet, specifically ResNet50, MobileNetV2, and EfficientNetB7, removing their fully connected layers to obtain feature representations.
  • Cross-Modal Attention and Fusion: A cross-modal attention mechanism is employed to allow the model to focus on the most relevant parts of both text and image. For fusing these attended features, they use both the Hadamard product (for element-wise interaction) and concatenation (to preserve modality-specific characteristics). The fused features are then passed through dense layers for final classification.

Datasets

The primary dataset used is the MDMD (Misogyny Detection Meme Dataset), provided by the DravidianLangTech@NAACL 2025 shared task. This dataset contains memes and their text transcriptions for both Tamil and Malayalam, annotated as either misogynistic or non-misogynistic.

  • Tamil Dataset: 1136 training, 284 validation, and 356 test samples.
  • Malayalam Dataset: 640 training, 160 validation, and 200 test samples.

The paper notes that both datasets are imbalanced, with a significantly larger number of non-misogynistic samples. Additionally, text transcriptions often contain code-mixed English, and images may have redundant elements like social media logos.

Key Insights and Ideas

  • Language-Specific Models Shine: The most striking finding is that language-specific BERT models (Tamil-BERT and Malayalam-BERT) fine-tuned on their respective languages, when paired with powerful CNN backbones like EfficientNet, significantly outperform general-purpose multilingual models (mBERT, Indic-BERT) for this task. This suggests that while multilingual models are useful, specialized models capture nuances better.
  • Multimodal Fusion is Key: The paper demonstrates that combining visual and textual information is crucial for accurately detecting misogynistic memes, outperforming single-modality approaches.
  • EfficientNet's Strength: The EfficientNetB7 architecture, when used as the CNN backbone, consistently yields better results across different BERT variants, highlighting its effectiveness in image feature extraction for this task.
  • Dataset Imbalance is a Challenge: The significant imbalance in the datasets, particularly for Tamil, directly impacts the F1 scores, with models struggling to correctly classify the minority (misogynistic) class.

Key Results

The authors achieved the following Micro-F1 scores on the test sets:

  • Tamil:
    • Best performing model: Tamil-BERT + EfficientNetB7 with an F1 score of 0.678.
  • Best multilingual model: mBERT + EfficientNetB7 with an F1 score of 0.644.
  • Malayalam:
    • Best performing model: Malayalam-BERT + EfficientNetB7 with an F1 score of 0.803.
  • Best multilingual model: mBERT + EfficientNetB7 with an F1 score of 0.742.

Overall, the paper reports an F1 score of 0.678 for Tamil memes and 0.803 for Malayalam memes using their best-performing multimodal configurations.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Labeled Data: The scarcity of labeled data impacts the model's ability to generalize to unseen instances.
  • Cultural and Linguistic Nuances: The approach may not fully capture the subtle cultural and linguistic specificities of Tamil and Malayalam that contribute to misogynistic expressions.
  • Inherited Biases: Pre-trained models can carry biases from their training data, which might affect classification accuracy.
  • Implicit/Sarcastic Expressions: The multimodal fusion process faces challenges in detecting implicit or sarcastic forms of misogyny.

For future work, they plan to:

  • Experiment with more advanced models, such as Vision Transformers (ViT).
  • Explore techniques to mitigate the dataset imbalance issue for improved performance.

Summary updated

BibTeX
@inproceedings{chowdhury2025fired,
  author = {Md. Sajid Alam Chowdhury and Mostak Mahmud Chowdhury and Anik Mahmud Shanto and Jidan Al Abrar and Hasan Murad},
  title = {{Fired\_from\_NLP@DravidianLangTech 2025: A Multimodal Approach for Detecting Misogynistic Content in Tamil and Malayalam Memes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure