taochen at SemEval-2022 Task 5: Multimodal Multitask Learning and Ensemble Learning

Chen Tao, Jung-jae KimNAACL 2022

TL;DR

This paper tackles the challenging task of identifying and classifying misogynistic content in online memes, which combine images and text. The authors propose a multimodal deep learning system that leverages multi-task learning to share knowledge between related subtasks and ensemble learning to combine the strengths of multiple models, achieving competitive results in the SemEval-2022 MAMI challenge.

What Problem Does This Paper Address?

Online memes, while often humorous, can also be potent vehicles for spreading harmful content, including misogyny. Identifying and classifying misogynistic memes is crucial for combating online hate speech. This task is particularly challenging because it requires understanding the interplay between visual and textual information within a meme, a multimodal problem. The SemEval-2022 Task 5 (MAMI) specifically addresses this by asking participants to not only detect if a meme is misogynistic but also to categorize the type of misogyny present (shaming, stereotype, objectification, violence). The paper highlights that existing multimodal classification methods face challenges in data representation, fusion, and co-learning, necessitating novel approaches.

Methods

The authors build their system upon the MMF (Multimodal Framework) codebase, which provides modular and scalable baseline models. Their approach involves several key strategies:

  • Multimodal Base Models: They utilize a suite of pre-trained multimodal models, including ConcatBOW/BERT/MMF, Late Fusion MMF, VILBERT, Visual BERT, and MMBT. These models are designed to process and fuse information from both image and text modalities. They also include a unimodal text model as a baseline for comparison.
  • Multi-task Learning (MTL): To address the correlated nature of the misogyny subtasks (Subtask A: misogynistic or not; Subtask B: type of misogyny), they adapt a Visual BERT model. They modify the decoder to have five separate classification layers, one for each output (Subtask A and the four types in Subtask B), allowing the model to learn shared representations and transfer knowledge across these related tasks.
  • Data Augmentation: To improve robustness, especially against blurry images or misclassified sarcasm, they augment the training data. This involves applying various transformations (rotation, noise, blur, flips, etc.) to existing memes, effectively increasing the dataset's diversity and helping the model generalize better.
  • Ensemble Learning: To further boost performance, they combine the predictions of multiple well-performing baseline models. They experiment with two ensemble strategies: a Multi-layer Perceptron (MLP) that takes output probabilities as input, and a majority voting layer.

Datasets

The paper uses the dataset provided for the SemEval-2022 Task 5 "Multimedia Automatic Misogyny Identification" (MAMI) challenge. This dataset consists of 10,000 memes with English captions for training and 1,000 memes for testing. The memes are in JPEG format, with captions provided in a separate text file. The task involves two subtasks: binary classification of misogyny (Subtask A) and multi-class categorization of misogyny types (Subtask B).

Key Insights and Ideas

  • Interdependence of Subtasks: The authors discover that the types of misogyny (Subtask B) might actually dictate the overall misogynistic classification (Subtask A), rather than the other way around. This insight led them to explore multi-task learning to capture these dependencies.
  • Early vs. Late Fusion: They observe that early fusion models (like Visual BERT) can be as competitive as late fusion models (like Concat BERT and MMBT), suggesting that modal correlations can be captured at different levels of feature extraction and decision-making.
  • Unimodal Text Dominance in Baselines: Interestingly, the unimodal text model performed better than the Concat BOW model, which uses simple bag-of-words for text representation. This highlights the importance of richer text representations and the contribution of visual information to understanding meme content.
  • Ensemble Strategy Matters: The choice of ensemble method (MLP vs. majority voting) and the number of models included (top-3 vs. top-6) significantly impacted performance, with different strategies proving optimal for Subtask A versus Subtask B.

Key Results

  • The system achieved 71.6% in Subtask A and 70.6% in Subtask B during the competition, ranking 4th and 6th on the leaderboard, respectively.
  • Multi-task learning on Visual BERT showed an improvement of 2.6% for Subtask B and 1.3% for Subtask A compared to single-task training.
  • The baseline models showed varying performance across subtasks, with Shaming and Violence being the most challenging to learn.
  • Data augmentation was found to be crucial, with the authors noting that around 30% of validation errors were due to blurry images and 17% to misclassified sarcasm.

Limitations and Future Work

  • Data Distribution Shift: The authors observed a discrepancy between model performance on the hold-out training data and the test data, suggesting a potential shift in data distribution. They propose generative multi-task learning as a future direction to address this.
  • External Information: The system could be improved by incorporating external knowledge, such as entity recognition and web-based information about celebrities or cultural references mentioned in memes.
  • Textual Nuances: Capturing meaning from text positions, sentence order in multi-frame memes, and handling anti-violence propaganda memes that use violent imagery but condemn violence are identified as areas for improvement.
  • Gender-Specific Language: Augmenting data by switching gender-referring words could help the model better understand memes that compare men and women.
  • Frame Layout and Cross-Subtask Dependency: Further research is suggested into incorporating frame layout in memes and explicitly modeling cross-subtask dependencies in multi-task learning.

Summary updated

BibTeX
@inproceedings{tao2022taochen,
  author = {Chen Tao and Jung-jae Kim},
  title = {{taochen at SemEval-2022 Task 5: Multimodal Multitask Learning and Ensemble Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure