INF-UFRGS at SemEval-2022 Task 5: analyzing the performance of multimodal models

Gustavo Lorentz, Viviane MoreiraNAACL 2022

TL;DR

This paper explores the effectiveness of five multimodal models (ViLBERT, VisualBERT, MMBT-Grid, ViLBERT CC, and VisualBERT COCO) for identifying misogynistic memes in the SemEval-2022 Task 5. While all models performed reasonably well, ViLBERT emerged as the top performer, highlighting the ongoing challenge of accurately detecting subtle forms of online hate speech.

What Problem Does This Paper Address?

Social media has unfortunately become a breeding ground for hateful speech, including misogyny. Identifying this content is crucial but challenging, especially when it's embedded in multimodal formats like memes, where text and images combine to convey a hateful message. Traditional text-only analysis methods fall short because the offensive meaning might be entirely dependent on the visual component or the interplay between text and image. This paper tackles the problem of automatically detecting misogynistic memes, aiming to improve upon existing methods that often require significant human effort.

Methods

The researchers experimented with five different multimodal models, all implemented within the MMF (Multimodal Framework) from Facebook AI Research. These models include:

  • MMBT-Grid: A supervised multimodal bi-transformer that fuses text and image embeddings.
  • ViLBERT and ViLBERT CC: These models use parallel visual and textual transformers with a novel "Co-attentional Transformer Layers" mechanism, allowing each modality to attend to the other.
  • VisualBERT and VisualBERT COCO: VisualBERT extends BERT by treating image region features as additional input tokens, enabling implicit alignment between text and image.

The key distinction between the "CC" and "COCO" versions and their unimodally pretrained counterparts lies in their pretraining strategy. The models were trained on the MAMI dataset using default hyperparameters, with a focus on macro-F1 score as the primary evaluation metric.

Datasets

The primary dataset used for training and evaluation was the MAMI (Multimedia Automatic Misogyny Identification) dataset provided for SemEval-2022 Task 5. This dataset consists of memes with English captions. The training data comprised 10,000 memes, with trial data having 100 memes and validation data having 1,000 memes. The paper also references the Hateful Memes Challenge dataset as a similar benchmark for multimodal hate speech detection.

Key Insights and Ideas

  • Multimodality is Key: The paper strongly emphasizes that understanding memes often requires considering both text and visuals simultaneously. A prime example illustrates how a seemingly innocuous text paired with an image can create a sexist message.
  • ViLBERT Shines: ViLBERT, a multimodal model, consistently outperformed others, suggesting its architecture and pretraining strategy are particularly effective for this task.
  • Pretraining Nuances: The study found that the distinction between unimodal and multimodal pretraining didn't lead to significant performance differences, which was somewhat surprising.
  • Image Features Matter: The use of image features during training seemed to have a notable impact, with models that didn't leverage them (like MMBT-Grid) performing worse.
  • Ensemble Limitations: Combining the predictions of multiple models through majority voting or averaging probabilities did not yield significant improvements over the best single model, indicating that the models might be making similar errors.
  • Subtle Language Clues: Analysis of misclassified memes revealed patterns related to specific words like "girl," "girls," "woman," and "women," suggesting that subtle linguistic cues, especially when combined with visuals, can be tricky for models to interpret correctly.

Key Results

  • All five models achieved Macro-F1 scores above 0.64.
  • ViLBERT was the best-performing model with a Macro-F1 score of 0.698, ranking 32nd out of 83 participants on the leaderboard.
  • MMBT-Grid was the worst-performing model with a Macro-F1 score of 0.649.
  • Despite MMBT-Grid's low overall score, it had a high True Positive (TP) rate (0.866), indicating it was good at identifying misogynistic memes but struggled with correctly classifying non-misogynistic ones (low True Negative (TN) rate).
  • VisualBERT COCO had the best TN rate (0.581) but also the highest False Negative (FN) rate (0.21).
  • The correlation between models was generally moderate to strong, explaining why ensemble methods didn't offer substantial gains.

Limitations and Future Work

The authors acknowledge that hyperparameter optimization was not extensively performed, suggesting that further tuning could potentially improve model performance. They also mention that the use of late/early fusion techniques, as suggested by prior work on similar challenges, could be explored as a future direction to enhance performance. Additionally, the analysis of misclassified memes points to the need for models to better understand subtle linguistic cues and their interplay with visual content.

Summary updated

BibTeX
@inproceedings{lorentz2022inf,
  author = {Gustavo Lorentz and Viviane Moreira},
  title = {{INF-UFRGS at SemEval-2022 Task 5: analyzing the performance of multimodal models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure