Codec at SemEval-2022 Task 5: Multi-Modal Multi-Transformer Misogynous Meme Classification Framework

Ahmed Mahran, Carlo Alessandro Borella, Konstantinos PerifanosNAACL 2022

TL;DR

This paper introduces a flexible, multi-modal framework for classifying misogynous memes, leveraging pre-trained transformer architectures. Their approach combines multiple encoders and a multi-task learning strategy to achieve strong performance on both binary and multi-label misogyny identification tasks.

What Problem Does This Paper Address?

Misogynous memes pose a significant challenge for automated detection due to their multi-modal nature (combining images and text), reliance on implicit cultural knowledge, and potential for misinterpretation. Existing methods often struggle to effectively fuse information from different modalities and generalize across various forms of misogyny. The SemEval-2022 Task 5 (MAMI) specifically targets this problem, requiring systems to identify misogynous memes and categorize the type of misogyny present.

Methods

The authors propose a generic framework built upon transformer architectures, aiming to be resource-efficient by utilizing pre-trained models. Their core strategies include:

  • Multi-Modal Embedding: They combine signals from different modalities (image and text) by feeding them into transformer encoders. They experiment with two main encoder architectures:
  • Shared Transformer Encoder: A single multi-layer transformer processes all input modalities after they've been embedded.
  • Multiple Transformers: Separate transformer encoders are used for each modality, with their outputs concatenated.
  • Pre-trained Encoders: For image encoding, they utilize CLIP and DETR, while for text, they use a BERT model pre-trained on hate speech data.
  • Multi-Task Learning: To leverage external datasets and improve generalization, they employ a multi-task learning approach, training on both the MAMI dataset and Facebook's hateful memes dataset.
  • Multi-Objective Optimization: Auxiliary objectives are used to regularize and fine-tune individual components of the system, aiming to improve overall performance.
  • Classification Strategies: They explore two classification approaches:
  • Pooled Sequence Embedding: The output of the [CLS] token from the transformer encoder is pooled and fed into a multi-head MLP classifier.
  • Transformer Decoder: The entire sequence embedding is processed by a transformer decoder, which learns class-specific query embeddings, followed by a single-head MLP classifier.

Datasets

  • MAMI Dataset: The primary dataset for the SemEval-2022 Task 5, containing memes with associated English text and labels for misogynous or not (Sub-task A) and types of misogyny (stereotype, shaming, objectification, violence) (Sub-task B).
  • Facebook Hateful Memes Dataset (FBHM): Used as an auxiliary dataset for multi-task learning, consisting of memes labeled as hateful or not.

Key Insights and Ideas

  • Leveraging Pre-trained Encoders is Crucial: The paper highlights the effectiveness of using pre-trained models like CLIP, DETR, and BERT, which significantly reduces the need for training from scratch and improves performance.
  • Multi-Modal Fusion Matters: Combining signals from both images and text is essential for accurately identifying misogynous memes, as the context often relies on the interplay between visual and textual elements.
  • Multi-Task Learning Boosts Performance: Incorporating external, related datasets (like FBHM) through multi-task learning demonstrably improves performance, suggesting that broader exposure to similar content aids generalization.
  • DETR's Competence: The authors found that DETR, while powerful for object detection, was less effective as an image encoder for this specific task compared to CLIP.
  • Class Query Embeddings Capture Semantics: Visualizations of the transformer decoder output show that learned class query embeddings tend to cluster around positive examples, acting as semantic centers for each misogyny type.

Key Results

  • Sub-task A (Binary Classification): Achieved a macro-average F1-Measure of 0.715 during the competition, and a post-evaluation score of 0.761.
  • Sub-task B (Multi-label Classification): Achieved a weighted-average F1-Measure of 0.698 during the competition.
  • Ablation Studies: The "multi-transformers encoders without output pooling" architecture variant, combined with CLIP and DETR backbones, emerged as a strong performer. Enabling projection alignment and contrastive embedding loss also showed slight improvements.

Limitations and Future Work

  • DETR Incompetence: The authors note that their use of DETR as an image encoder was "incompetent" compared to CLIP, suggesting potential areas for improvement in how DETR features are extracted or utilized.
  • More Experiments Needed: The authors acknowledge the need for more experiments with different backbone architectures and datasets to further validate their approach.
  • Exploring More Objectives: They suggest exploring additional objectives and regularization techniques, potentially inspired by their visualization of class query embeddings, to further enhance model performance. For instance, making positive instances' decoder outputs closer to their corresponding class query embeddings.

Summary updated

BibTeX
@inproceedings{mahran2022codec,
  author = {Ahmed Mahran and Carlo Alessandro Borella and Konstantinos Perifanos},
  title = {{Codec at SemEval-2022 Task 5: Multi-Modal Multi-Transformer Misogynous Meme Classification Framework}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure