UAEM-ITAM at SemEval-2022 Task 5: Vision-Language Approach to Recognize Misogynous Content in Memes

Edgar Roman-Rangel, Jorge Fuentes-Pacheco, Jorge Hermosillo ValadezNAACL 2022

TL;DR

This paper presents a multimodal approach for identifying misogynous content in memes by combining visual and textual features. Their framework leverages pre-trained vision-language models and BERT for rich representations, achieving competitive results on the MAMI 2022 challenge.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically identifying misogynous content within internet memes, which often combine images and text in complex ways. The MAMI 2022 competition specifically aimed to address this by organizing tasks around identifying misogynous memes (Sub-task A) and categorizing the type of misogyny (Sub-task B), such as stereotyping, shaming, objectification, and violence. This is a difficult problem because misogyny can be subtle, context-dependent, and expressed through the interplay of visual and textual elements, making simple keyword matching or image analysis insufficient.

Methods

The authors propose a multimodal framework that extracts features from both the image and text components of a meme at both global and local levels.

  • Image Descriptors:
    • Global: A pre-trained InceptionV3 CNN, fine-tuned on the MAMI dataset, generates a 64-dimensional global image representation.
  • Local: A pre-trained VinVL model is used to detect and describe individual objects within the image, yielding 2048-dimensional feature vectors per object. Principal Component Analysis (PCA) is then applied to reduce these local features to an 8-dimensional representation per object, and then further reduced to 64 dimensions per image.
  • Text Representations:
    • Contextual Embeddings: A pre-trained small uncased BERT model generates a 512-dimensional sentence embedding. This is then fine-tuned using a multi-layer perceptron (MLP).
  • Enhanced Vocabulary Representations: For each detected object in the image, VinVL provides lists of nouns and adjectives. These words are concatenated into a single vector and used to train a separate classification network, similar in architecture to the contextual embedding model.

Finally, the 64-dimensional global image descriptor, the 64-dimensional local image descriptor, the 512-dimensional contextual text embedding, and the 512-dimensional enhanced vocabulary representation are concatenated to form a 256-dimensional multimodal feature vector. This vector is then fed into a final MLP classifier with nine fully-connected layers to predict one of the five misogyny categories.

Datasets

The primary dataset used is the MAMI 2022 challenge dataset. The models are trained and evaluated on this dataset. The paper does not mention the creation of any new benchmarks.

Key Insights and Ideas

  • Multimodal Fusion is Crucial: The success of the approach hinges on effectively combining information from both visual and textual modalities, recognizing that memes derive their meaning from their interplay.
  • Local Descriptors Capture Nuance: Using local object detection and description for images, and enhanced vocabulary for text, allows the model to capture finer-grained details that might be missed by global representations alone.
  • Transfer Learning is Key: Leveraging powerful pre-trained models like InceptionV3, BERT, and VinVL significantly boosts performance, as these models have already learned rich representations from massive datasets.
  • Class Imbalance Challenges Performance: The paper highlights that for less frequent or unbalanced classes (like "shaming" and "violence"), performance is lower due to limited training examples, which hinders fine-tuning.

Key Results

The proposed model achieved competitive results on Sub-task B of the MAMI 2022 challenge:

  • Performance: The model's F1 score on the test set was 0.646, which is 4% better than the baseline provided by the organizers.
  • Comparison: It fell behind the best-performing model by 12%.
  • Class Performance: The "misogynous" class showed the highest accuracy (0.97), while "stereotype" had the lowest (0.90). For F1 scores, "shaming" and "violence" had the worst performance (0.68 and 0.71 respectively), attributed to class imbalance. Balanced classes like "misogynous," "stereotype," and "objectification" performed better.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Class Imbalance: The performance on unbalanced classes is a significant limitation.
  • Future Work Suggestions:
    1. Fine-tuning Classification Thresholds: Independently adjusting the sigmoid output layer's threshold for each class.
  • Optimizing Loss Function: Directly optimizing the loss function for the F1 score.
  • Weighting Class Contributions: Adjusting the loss function to give more weight to underperforming classes.

Summary updated

BibTeX
@inproceedings{romanrangel2022uaem,
  author = {Edgar Roman-Rangel and Jorge Fuentes-Pacheco and Jorge Hermosillo Valadez},
  title = {{UAEM-ITAM at SemEval-2022 Task 5: Vision-Language Approach to Recognize Misogynous Content in Memes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure