Logically at the Constraint 2022: Multimodal role labelling

Ludovic Kun, Jayesh Bankoti, David KiskovskiACL 2022

TL;DR

This paper presents a winning system for the CONSTRAINT 2022 challenge, which aims to identify the roles (Hero, Villain, Victim, Other) of entities within memes. Their approach, an ensemble of multimodal and unimodal models, achieved the top rank by effectively combining text and image information to understand the nuanced sentiment and context of memes.

What Problem Does This Paper Address?

The proliferation of online content, particularly memes, presents a significant challenge for understanding and combating harmful narratives. Memes, with their combination of images and text, are complex to analyze, and traditional methods often focus solely on text. This paper tackles the problem of multimodal role labelling for memes, specifically identifying how entities within a meme are portrayed – whether glorified (Hero), vilified (Villain), victimized (Victim), or otherwise (Other). This task is crucial for social media analysis and content moderation, as it goes beyond simple sentiment analysis to understand the specific narrative roles entities play.

Methods

The authors employed a multi-class classification approach, treating the task as assigning one of four labels to each entity within a meme. Their core strategy is an ensemble method, combining the predictions of multiple individual models to improve robustness and performance.

The individual models fall into two categories:

  • Unimodal Models: These models process either text (OCR of the meme) or image data independently. They experimented with transformer-based models like DeBERTa and RoBERTa for text processing.
  • Multimodal Models: These models integrate both text and image information. They explored several architectures:
  • Naive Merging: Extracting features from text (DeBERTa) and images (EfficientNet-B7) separately and then concatenating them before feeding them into a classifier.
  • Pre-trained Multimodal Transformers: Utilizing models like MMBT and VisualBERT, which are designed to jointly process text and images.
  • ViLT (Vision and Language Transformers): A more streamlined architecture that processes text and visual features without separate embeddings.

A key aspect of their multimodal approach involved meta-data extraction from images. This included:

  • Celebrity Detector: Identifying famous faces in memes, even if they weren't explicitly mentioned in the text, to infer their role.
  • Sub Image Detector: Using YOLOv5 to identify multiple distinct images within a single meme, which can convey contrasting messages.

The final winning model was an ensemble of 5 classifiers, achieved by averaging the predictions of selected unimodal and multimodal models.

Datasets

The primary dataset used for training and evaluation was the CONSTRAINT 2022 dataset. This dataset consists of memes related to US politics and COVID-19, with associated OCR text and entity annotations. The authors note that the dataset is relatively small and imbalanced, with the "Other" class comprising over 75% of entities.

Key Insights and Ideas

  • Multimodality is Crucial for Memes: The paper strongly emphasizes that understanding memes requires processing both text and images. Relying solely on OCR is insufficient, as visual cues and implied meanings are vital for accurate role labelling.
  • Ensembling for Robustness: Combining multiple models, especially diverse ones, significantly boosts performance, particularly in low-resource settings where individual models might overfit.
  • Meta-Data Extraction Enhances Understanding: The inclusion of a celebrity detector and sub-image detection proved valuable for inferring entity roles when explicit mentions were absent or ambiguous.
  • Transformer Dominance: Transformer-based models, particularly DeBERTa, showed strong performance in unimodal text classification.
  • Challenges of Low-Resource Multimodal Learning: Pre-trained multimodal models, while promising, underperformed when the pre-training data distribution differed significantly from the target task, highlighting the need for task-specific fine-tuning or transfer learning strategies.

Key Results

The authors' ensemble model achieved the top rank on the CONSTRAINT 2022 leaderboard, obtaining a macro F1-score of 0.58 on the test set. This performance surpassed all other competition systems and baseline models. They also reported that their best performing model combined the averaging of ViLT, RoBERTa-large, DeBERTa-large, naive multimodal, and DeBERTa-xlarge models.

Limitations and Future Work

The authors acknowledge several limitations:

  • Low-Resource Data: The small size and imbalance of the dataset were significant challenges, limiting the ability to train robust models from scratch and potentially leading to overfitting.
  • Entity Ambiguity: Entities were not always present in the image or text, requiring contextual understanding that is difficult to automate.
  • OCR Imperfections: The OCR process sometimes resulted in poorly clustered text or spelling mistakes, impacting text-based analysis.

For future work, they suggest that a more elaborate and extensive dataset would be beneficial for benchmarking and advancing research in this area. They also highlight the potential for further exploration of multimodal inference problems like role labelling.

Summary updated

BibTeX
@inproceedings{kun2022logically,
  author = {Ludovic Kun and Jayesh Bankoti and David Kiskovski},
  title = {{Logically at the Constraint 2022: Multimodal role labelling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure