R2D2 at SemEval-2022 Task 5: Attention is only as good as its Values! A multimodal system for identifying misogynist memes

Mayukh Sharma, Ilanthenral Kandasamy, Vasantha W BNAACL 2022

TL;DR

This paper presents a multimodal deep learning system for identifying misogynistic memes, a challenging task due to the interplay of images and text. Their approach leverages pre-trained models for feature extraction and employs an attention mechanism to fuse these modalities, achieving a strong performance and ranking in the SemEval-2022 Task 5 challenge.

What Problem Does This Paper Address?

The internet, particularly social media, has become a breeding ground for hate speech, with misogyny—prejudice or hate against women—seeing a significant rise. Internet memes, a popular form of online communication, are uniquely challenging to analyze because they combine images and text, often in ways that can be harmless individually but become misogynistic when juxtaposed. While previous work has focused on identifying misogynous text, there's a critical need to address this phenomenon in multimodal content like memes. SemEval-2022 Task 5: MAMI (Multimedia Automatic Misogyny Identification) was designed to tackle this problem by providing a dataset of misogynous memes and their types.

Methods

The R2D2 system employs a multimodal approach, combining visual and textual information to identify misogynous memes. They utilize pre-trained models as feature extractors for both modalities: Vision Transformer (ViT) for images and various transformer-based language models (BERT, RoBERTa, MPNet, DeBERTa) for text.

The core of their multimodal fusion strategy lies in an attention mechanism, specifically scaled dot-product attention. This mechanism allows the model to learn how to weigh the importance of different parts of the image and text when forming a joint representation. A key aspect of their work is exploring the role of the "query" in this attention mechanism. They found that using image features as the query and text features as the "keys" and "values" yielded superior results, suggesting that this configuration effectively preserves textual information while allowing the image to guide the fusion process.

For classification, they use dense layers followed by a sigmoid activation for binary classification (Subtask A: misogynous vs. non-misogynous) and a multi-branch architecture for multi-label classification of misogyny types (Subtask B: stereotype, shaming, objectification, violence). They also experimented with simpler feature concatenation as a baseline fusion method.

Datasets

The research utilizes the dataset provided for SemEval-2022 Task 5: MAMI. This dataset consists of internet memes with associated text, labeled for misogyny detection (Subtask A) and for specific types of misogyny (Subtask B). The paper provides statistics for the dataset split into training, validation, and test sets for both subtasks.

Key Insights and Ideas

  • The "Query" Matters in Multimodal Attention: The choice of which modality serves as the "query" in the attention mechanism significantly impacts performance. Using image features as the query for attention proved more effective than using text features, suggesting a nuanced interaction between modalities in understanding memes.
  • Textual Dominance in Misogyny Detection: Despite the multimodal nature of memes, the study found that text-based models generally outperformed image-based models, and even multimodal approaches sometimes only slightly improved upon strong text-only baselines. This indicates that the textual content often carries a significant portion of the misogynistic signal.
  • Attention as a Powerful Fusion Tool: The attention-based fusion mechanism demonstrated its effectiveness in capturing the complex interplay between image and text, outperforming simple concatenation in several configurations.
  • Hierarchical Approach for Subtask B: For identifying the type of misogyny, a hierarchical approach was employed, leveraging the predictions from the misogyny detection model (Subtask A) to inform the classification of misogyny types.

Key Results

  • Subtask A (Misogynous Meme Identification): The best-performing system, a voting ensemble of attention-based multimodal models with image features as query, achieved an F1 score of 0.757, ranking 3rd on the leaderboard.
  • Subtask B (Misogyny Type Identification): The BERT + ViT model using attention with image features as query achieved an F1 score of 0.690, ranking 10th on the leaderboard.
  • Unimodal text models generally performed better than unimodal image models.
  • The attention mechanism with image features as query consistently outperformed concatenation and unimodal baselines.

Limitations and Future Work

  • Textual Dominance: While the multimodal approach is crucial, the results suggest that textual features are currently more dominant in identifying misogyny in memes. Future work could explore ways to better leverage visual cues.
  • Non-Commutative Nature of Attention: The paper highlights that attention is not commutative, meaning the order and role of modalities (query, key, value) are critical. Further research is suggested to explore how attention mechanisms learn multimodal features and to compare the roles of individual modalities more deeply.
  • Future Exploration of Attention: The authors express interest in further investigating how attention mechanisms learn multimodal features and comparing the contributions of different modalities in multimodal tasks.

Summary updated

BibTeX
@inproceedings{sharma2022r,
  author = {Mayukh Sharma and Ilanthenral Kandasamy and Vasantha W B},
  title = {{R2D2 at SemEval-2022 Task 5: Attention is only as good as its Values! A multimodal system for identifying misogynist memes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure