ASRtrans at SemEval-2022 Task 5: Transformer-based Models for Meme Classification

Ailneni Rakshitha Rao, Arjun RaoNAACL 2022

TL;DR

This paper tackles the challenging task of automatically identifying misogynistic content in memes, which combine images and text. The authors developed a system using transformer-based models, achieving impressive results by fine-tuning pre-trained language models with external sexism-related data and employing an ensemble of multimodal models.

What Problem Does This Paper Address?

Social media has become a breeding ground for hate speech and misogyny, with memes being a particularly potent vehicle for such harmful content. Identifying misogyny in memes is complex because it requires understanding the interplay between visual and textual elements, often in nuanced and context-dependent ways. SemEval-2022 Task 5, the Multimedia Automatic Misogyny Identification (MAMI) challenge, specifically aimed to address this by classifying memes into categories like stereotyping, shaming, objectification, and violence. This task is crucial for developing tools to combat online harassment and protect vulnerable groups, especially women, who are disproportionately targeted.

Methods

The core of the ASRtrans system relies on transformer-based architectures, which have become state-of-the-art in natural language processing due to their ability to capture rich contextual embeddings. The authors explored two main avenues:

  • Text-based Pre-training: They fine-tuned a pre-trained RoBERTa-base model. Crucially, they further trained this model on a curated dataset of external sexism-related text instances (from sources like workplace comments and sitcoms) before fine-tuning it on the MAMI task data. This "tuned RoBERTa" (tRoBERTa) approach proved highly effective.
  • Multimodal Pre-training: They also investigated established multimodal models like VisualBERT and MMBT. VisualBERT integrates BERT with visual features from an object detector, while MMBT combines large language models with convolutional neural networks.
  • Ensemble Learning: To further boost performance, they combined the predictions of their best text-based model (tRoBERTa) and a multimodal model (VisualBERT) using a weighted average ensemble.

Datasets

The primary dataset used was from SemEval-2022 Task 5 (MAMI), consisting of 10,000 memes with associated text transcriptions and labels. This dataset is divided into two sub-tasks:

  • Sub-task A: Binary classification (misogynous or not).
  • Sub-task B: Multi-label classification (categorizing into stereotype, shaming, objectification, and violence).

For augmenting their text-based model, they curated a dataset of approximately 5,049 sexist text instances from various sources, including:

  • Grosz and CONDE-CESPEDES (2020) dataset of sexist workplace comments.
  • Singh et al. (2021) dataset of sexism in English sitcoms.
  • EXIST2021 dataset (Rodríguez-Sánchez et al., 2021-09) of sexist text instances.

Key Insights and Ideas

  • The Power of Task-Specific Fine-tuning: Simply using pre-trained models wasn't enough. Further training RoBERTa on a dataset specifically curated for sexism significantly improved its performance on the MAMI task. This highlights the importance of domain-specific adaptation.
  • Textual Nuance is Key (and Tricky): The paper points out a significant challenge in correctly classifying memes that contain negations (e.g., "not misogynous"). Language models, even advanced ones, can struggle with these subtle linguistic cues, leading to misclassifications.
  • Multimodal Ensembling Pays Off: Combining the strengths of different models, particularly a strong text-based model and a multimodal one, yielded a noticeable performance boost, demonstrating the value of diverse perspectives in tackling complex tasks.
  • Single Image Embedding Suffices: For the MMBT model, using a single image embedding provided better results than multiple embeddings, suggesting that the way features are aggregated is more critical than the sheer number of embeddings.

Key Results

The ASRtrans system achieved highly competitive results in the SemEval-2022 Task 5:

  • Sub-task A (Binary Classification): Ranked 3rd out of 83 teams with an F1-macro score of 0.761. This significantly outperformed the official baselines.
  • Sub-task B (Multi-label Classification): Ranked 7th out of 48 teams with a weighted F1 score of 0.705.

The paper also shows that their tuned RoBERTa model, when combined with features from ResNet-152, performed slightly better than VisualBERT without multimodal pre-training. An ensemble of these two models further improved performance by approximately 1% in F1 score for sub-task A and 0.5% for sub-task B.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Handling Negation: The model's difficulty in understanding negation is a significant limitation, leading to misclassifications in memes where "not" plays a crucial role. Future work should focus on developing methods to better handle such linguistic nuances.
  • Exploring Other Multi-label Techniques: The paper mentions exploring other problem transformation approaches for multi-label classification, such as Classifier Chains, to better model label dependencies and hierarchies.
  • Training Multimodal Models from Scratch: The authors plan to investigate training VisualBERT from scratch on meme datasets to see if this yields further improvements compared to fine-tuning a model pre-trained on general datasets like COCO.
  • Robustness to Missing Modalities: The approach of fine-tuning a text model with external data is noted as being computationally efficient and flexible, making it potentially applicable to scenarios where one modality might be missing.

Summary updated

BibTeX
@inproceedings{rao2022asrtrans,
  author = {Ailneni Rakshitha Rao and Arjun Rao},
  title = {{ASRtrans at SemEval-2022 Task 5: Transformer-based Models for Meme Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure