QiNiAn at SemEval-2022 Task 5: Multi-Modal Misogyny Detection and Classification

Qin Gu, Nino Meisinger, Anna-Katharina DickNAACL 2022

TL;DR

This paper presents two models for detecting and classifying misogynistic memes, tackling the challenge of combining visual and textual information. Their ensemble model for binary classification achieved a strong 5th place, while a RoBERTa-based model for multi-label classification of misogyny types secured 19th place, demonstrating the effectiveness of their approaches in this complex task.

What Problem Does This Paper Address?

The internet, particularly male-dominated spaces like microblogging sites, often hosts misogynistic content in the form of jokes and memes. This content can perpetuate harmful stereotypes and ideas about women. Detecting this misogyny is challenging because it's often multi-modal, relying on the interplay between images and text. A meme might seem harmless when viewed in isolation, but the combination of visual and textual elements can reveal its misogynistic nature. The SemEval-2022 Task 5 (MAMI challenge) aimed to address this by creating a benchmark for automatically detecting and classifying misogynistic memes.

Methods

The authors developed two distinct models for the two subtasks of the challenge:

  • Subtask A: Binary Misogyny Detection: They employed an ensemble approach for classifying memes as either misogynistic or not. This model combines predictions from four "base" classifiers: two text-based (Multinomial Naive Bayes and Gradient Boosting, both using TF-IDF vectors) and two image-based (Random Forest using handcrafted features). The handcrafted image features included Hu moment invariants, Haralick textures, and image histograms, aiming to capture structural and textural information. Additionally, they incorporated several novel features derived from the images and text: the count and average age of depicted men/women, a binary nudity score, and a sentiment score of the text. These features were fed into a separate Multinomial Naive Bayes classifier, and its predictions were combined with the other classifiers via majority voting.
  • Subtask B: Multi-label Classification of Misogyny Types: For this task, which involved classifying memes into specific categories of misogyny (stereotypes, shaming, objectification, violence), they opted for a purely text-based approach using a transformer model. Specifically, they utilized a pre-trained RoBERTa model from the Simple Transformers library. This model was adapted for multi-label classification by having five output neurons, one for each of the five labels (misogyny, shaming, stereotype, objectification, and violence).

Datasets

The study used the dataset provided for the SemEval-2022 Task 5 (MAMI challenge). This dataset includes both images and text transcriptions of memes. For Subtask A, the training set contained 10,000 memes, with an equal split between misogynistic and harmless examples. For Subtask B, the training set also included labels for four specific categories of misogyny: stereotypes (2810 memes), objectification (2202), shaming (1274), and violence (953). The authors note that many misogynistic memes fall into multiple categories.

Key Insights and Ideas

  • Ensemble Power for Multi-modal: The success of their ensemble model for binary classification highlights that combining diverse classifiers, even traditional ones with TF-IDF features, can be highly effective for multi-modal tasks.
  • Nudity as a Strong Indicator: The nudity score proved to be a significant feature for detecting misogynistic memes, with a notable difference in explicit content between misogynistic and non-misogynistic examples.
  • Limited Utility of Other Handcrafted Features: Surprisingly, features like sentiment score, and the count/average age of depicted individuals, provided less discriminative power than expected. This suggests that subtle forms of misogyny might not be easily captured by these specific handcrafted features.
  • Transformer Dominance for Text-based Classification: For the more nuanced task of classifying misogyny types, a RoBERTa-based transformer model, even when used solely on text, outperformed other approaches, underscoring the power of large pre-trained language models.
  • Image Information's Limited Impact on Subtask B: Despite the task being multi-modal, their RoBERTa model for Subtask B, which excluded image information, performed well. This suggests that for certain types of misogyny, the textual cues might be sufficiently strong, or that effectively integrating image information for these specific categories remains a challenge.

Key Results

  • Subtask A: Their ensemble model achieved 5th place on the leaderboard with a macro F1-score of 0.665. The gradual build-up of the ensemble showed that Gradient Boosting and Random Forest classifiers provided substantial performance gains.
  • Subtask B: The RoBERTa-based model achieved 19th place with a weighted F1-score of 0.637. While the model performed reasonably well, it produced more false negatives than positives for most categories, except shaming. Approximately a third of the images were correctly identified across all four categorization labels, with 731 memes having at least three labels matching.

Limitations and Future Work

  • Limitations:
    • The handcrafted features (sentiment, age, gender counts) showed limited utility for binary classification.
  • For Subtask B, the model struggled with subtle misogyny that didn't rely on explicit visual cues or easily detectable text. Memes with minimalistic drawings, low quality, or subtle references (like handprints) were harder to classify.
  • The multi-modal model for Subtask B, which incorporated image information, did not perform as well as the text-only RoBERTa model.
  • The model for Subtask B produced more false negatives than positives, suggesting a need for a stricter classifier for certain categories.
  • Future Work:
    • Explore how well similar ensemble models, potentially with more powerful text classifiers like BERT, perform.
  • Investigate further how to effectively integrate image information into transformer-based models for multi-label misogyny classification.
  • Experiment with different weighting schemes and threshold values for the RoBERTa model in Subtask B.
  • Develop better multi-modal models that can capture subtle misogynistic cues.

Summary updated

BibTeX
@inproceedings{gu2022qinian,
  author = {Qin Gu and Nino Meisinger and Anna-Katharina Dick},
  title = {{QiNiAn at SemEval-2022 Task 5: Multi-Modal Misogyny Detection and Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure