CUET-NLP@DravidianLangTech-ACL2022: Investigating Deep Learning Techniques to Detect Multimodal Troll Memes

Md Hasan, Nusratul Jannat, Eftekhar Hossain, Omar Sharif, Mohammed Moshiul HoqueACL 2022

TL;DR

This paper tackles the challenging task of automatically detecting "troll memes" in Tamil, which combine images and text to spread offensive content. The researchers explored various deep learning models, finding that while textual analysis alone performed well, multimodal approaches didn't significantly improve results, suggesting a need for more sophisticated fusion techniques.

What Problem Does This Paper Address?

The internet has become a primary channel for communication, and memes, with their blend of visuals and text, are a popular way to convey ideas and emotions. However, this format is also exploited to spread hate speech, trolling, and offensive content. Detecting these "troll memes" is particularly difficult due to their multimodal nature (image + text) and the region-specific interpretations of language and cultural references. This paper focuses on this problem within the Tamil language, aiming to develop effective deep learning models for classifying troll memes.

Methods

The researchers employed a range of deep learning techniques to tackle multimodal troll meme detection. Their approach involved:

  • Data Preprocessing: Standard text cleaning (removing symbols, punctuation) and image resizing to a uniform (224x224x3) format. Text was converted into numerical vectors using a Keras tokenizer and padding.
  • Visual Feature Extraction: Pre-trained Convolutional Neural Networks (CNNs) like VGG16, ResNet50, and VGG19 were used for image feature extraction, leveraging transfer learning by freezing the top layers and adding a global average pooling and sigmoid classification layer.
  • Textual Feature Extraction: Various deep learning architectures were explored for text analysis, including CNNs and Recurrent Neural Networks (RNNs) like LSTMs. Specifically, they implemented CNN models and a combined LSTM+CNN model to capture long-term dependencies in text.
  • Multimodal Fusion: Features extracted from both visual and textual models were combined. They experimented with concatenating the output layers of the visual and textual models, followed by a final sigmoid layer for prediction.

Datasets

The study utilized a dataset provided for the CUET-NLP@DravidianLangTech-ACL2022 shared task. This dataset contained Tamil memes labeled as either "troll" or "not-troll," with each meme consisting of an image and an associated caption. Participants could use the image, caption, or both for classification. The dataset was split into training and testing sets, with 2300 samples for training and 667 for testing.

Key Insights and Ideas

  • Textual Dominance: Surprisingly, textual analysis alone often outperformed multimodal approaches in terms of the weighted f1-score. This suggests that for this specific dataset and task, the textual content carried more discriminative power for identifying troll memes.
  • Multimodal Fusion Challenges: Simple concatenation of features from visual and textual models did not yield significant improvements over the best textual models. This highlights the complexity of effectively merging information from different modalities, especially when one modality might be more informative than the other.
  • Dataset Characteristics: The authors noted that the dataset might contain memes with similar visual content but different captions, or images that don't convey explicit information, making multimodal analysis challenging. This could explain why some multimodal models struggled.
  • Bias Towards Troll Memes: Error analysis revealed that all models exhibited a bias towards classifying memes as "troll," incorrectly labeling a significant portion of non-troll memes. This suggests a need for more balanced datasets or specialized techniques to mitigate this bias.

Key Results

  • The best performing textual model was LSTM+CNN, achieving a weighted f1-score of 0.52.
  • Among visual models, VGG16 and VGG19 achieved the highest weighted f1-score of 0.50.
  • The best multimodal model was CNN-Text+VGG16, which achieved a weighted f1-score of 0.49.
  • The team's overall best model, CNN+LSTM+VGG16, achieved a weighted f1-score of 0.46, placing them fourth in the shared task competition.

Limitations and Future Work

  • Limitations:
    • Simple feature concatenation for multimodal fusion did not yield expected improvements.
  • Models showed a bias towards classifying memes as trolls, leading to misclassification of non-troll memes.
  • Some memes had missing captions, hindering textual and multimodal analysis.
  • Future Work:
    • Investigate more advanced multimodal fusion techniques.
  • Explore transformer-based models (e.g., Vision Transformer, IndicBERT, mBERT, MuRIL) for both visual and textual feature extraction.
  • Utilize an extended dataset to potentially improve performance and address biases.

Summary updated

BibTeX
@inproceedings{hasan2022cuet,
  author = {Md Hasan and Nusratul Jannat and Eftekhar Hossain and Omar Sharif and Mohammed Moshiul Hoque},
  title = {{CUET-NLP@DravidianLangTech-ACL2022: Investigating Deep Learning Techniques to Detect Multimodal Troll Memes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure