AMS_ADRN at SemEval-2022 Task 5: A Suitable Image-text Multimodal Joint Modeling Method for Multi-task Misogyny Identification

Da Li, Ming Yi, Yukai HeNAACL 2022

TL;DR

This paper introduces two novel multimodal deep learning models for identifying misogyny in image-text posts, crucial for maintaining healthy online environments. Their approach combines text and image analysis, achieving competitive results in a SemEval task by employing a multi-stage training framework and a model ensemble with post-processing.

What Problem Does This Paper Address?

The internet, particularly image-based social media like Twitter and Instagram, has become a significant platform for women's voices. However, it also amplifies gender discrimination and aggressive content, perpetuating harmful stereotypes and inequality. Identifying and filtering such misogynistic content is essential for a healthy online social environment. This paper tackles the challenge of automatically detecting misogyny in multimedia posts, which often combine images and text, making it a complex task that requires understanding the interplay between visual and textual cues.

Methods

The authors propose two main multimodal model architectures:

  • Transformer Single-flow Structure: This model processes image and text features together in a single stream of transformer modules. The idea is to map multi-modal features into a shared vector space. They leverage pre-trained VisualBERT weights and combine image features from ResNet-152, EfficientNet-b2, EfficientNet-b4, and EfficientNet-b7 with text features from BERT-base-uncased.
  • Double-tower Structure: This architecture uses two parallel models, one for image and one for text, with MLPs at the output for joint training. This approach is less reliant on extensive pre-training and is more flexible, making it suitable for tasks with limited training data. They use ResNet-18 for image features and Bertweet for text.

To further improve performance, they employ:

  • Model Ensemble: A weighted average of the predictions from the single-flow and double-tower models.
  • Three-stage Training Framework:
    1. Multi-task Binary Classification: Initially trains to predict misogyny, stereotype, shaming, objectification, and violence.
  • Single-task Binary Classification: Uses external data to train a binary classifier for misogyny, specifically to augment the training set for Sub-task A.
  • Post-processing: Refines predictions by using the misogyny prediction to correct classifications for other categories, recognizing that certain misogynistic sub-types imply overall misogyny.

Datasets

  • MAMI Dataset (SemEval-2022 Task 5): This is the primary dataset used for training and evaluation. It consists of image-text posts.
  • Sub-task A: Focuses on binary misogyny identification.
  • Sub-task B: Focuses on identifying specific types of misogyny (stereotype, shaming, objectification, violence).
  • External Data: Used to augment the training set for Sub-task A, specifically from the "Memotion task" (Sharma et al., 2020).

Key Insights and Ideas

  • Multimodal Fusion is Crucial: Misogyny often relies on the interplay between images and text. Simple text-only or image-only models would likely miss many instances.
  • Double-tower Architecture's Adaptability: For tasks with limited training data, the double-tower architecture, which learns cross-modal features directly from the task data, can outperform single-flow models that heavily rely on extensive pre-training on different domains.
  • Pre-training Domain Mismatch: Standard image-text pre-training (e.g., on ImageNet, COCO) might not be optimal for social media text-image data where text is often OCR'd from images and carries emotional context.
  • Post-processing for Refinement: A simple post-processing step, leveraging the relationship between overall misogyny and its sub-types, can significantly improve performance, especially for categories like shaming or stereotype.
  • Humor as a Misdirection: Some misogynistic content is disguised as humor, making it harder to detect and leading to misclassifications.

Key Results

The proposed methods achieved competitive results in the SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification.

  • Sub-task A: Ranked 12th with a macro F1-score of 0.746.
  • Sub-task B: Ranked 10th with a macro F1-score of 0.706.
  • The model ensemble with post-processing achieved the best results, outperforming the organizers' baseline by significant margins. Specifically, the post-processing method achieved a macro F1 of 0.746 for Sub-task A and 0.708 for Sub-task B.
  • The double-tower model generally performed better than the single-flow model, especially without extensive domain-specific pre-training.

Limitations and Future Work

  • Limitations:
    • The single-flow model's performance is heavily dependent on pre-training, and standard pre-training tasks might not be ideal for social media content.
  • Some misogynistic samples are subtle and difficult to detect, even with multimodal approaches, especially when humor is involved.
  • The paper notes that the effect of defining Sub-task B as a single-task binary classification was not significantly better than multi-task binary classification, suggesting potential complexities in that approach.
  • Future Work:
    • Explore more unlabeled sample data for pre-training the transformer single-flow method.
  • Develop better multimodal methods specifically for Twitter scenarios.
  • Investigate how to better handle subtle misogyny and humor-disguised misogyny.

Summary updated

BibTeX
@inproceedings{li2022ams,
  author = {Da Li and Ming Yi and Yukai He},
  title = {{AMS\_ADRN at SemEval-2022 Task 5: A Suitable Image-text Multimodal Joint Modeling Method for Multi-task Misogyny Identification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure