LT3 at SemEval-2021 Task 6: Using Multi-Modal Compact Bilinear Pooling to Combine Visual and Textual Understanding in Memes

Pranaydeep Singh, Els LefeverACL 2021

TL;DR

This paper tackles the challenge of automatically detecting persuasion techniques in internet memes, which often rely on a combination of text and images. The authors propose a multi-modal system that effectively combines visual and textual features using "memebeddings" and Multi-modal Compact Bilinear (MCB) Pooling, achieving strong results in a SemEval task.

What Problem Does This Paper Address?

Internet memes have become a powerful and ubiquitous tool for communication, often used to spread disinformation and propaganda. Unlike traditional text-based propaganda, memes leverage both visual and textual elements, making their analysis more complex. Automatically identifying the persuasive techniques employed in these multi-modal artifacts is crucial for understanding and combating online disinformation campaigns. Previous work on propaganda detection primarily focused on text, leaving a gap in analyzing the unique challenges posed by memes. This paper addresses this gap by proposing a system specifically designed to understand and classify persuasion techniques within memes by integrating both visual and textual information.

Methods

The proposed system, named LT3, is a multi-modal approach designed for the SemEval-2021 Task 6, which involves identifying 22 different persuasion techniques in memes. The system comprises three main sub-modules:

  1. Visual Pre-processor: A ResNet-51 architecture, pre-trained to classify memes into one of 6200 Reddit subreddits. The hypothesis is that learning to categorize memes by their source subreddit implicitly teaches the model to recognize visual elements associated with different genres or emotions.
  2. Text Pre-processor: A pre-trained BERT-large-uncased model, fine-tuned on the PTC Corpus (used in SemEval-2020 Task 11 for propaganda detection). This leverages the power of BERT for understanding textual nuances.
  3. Integration Network: This is the core of the novelty. The embeddings from the visual and text pre-processors are combined using Multi-modal Compact Bilinear (MCB) Pooling. MCB Pooling is a technique that efficiently combines features from different modalities by constructing count sketch projections and then performing a convolution. This method aims to capture higher-order interactions between visual and textual features, overcoming the limitations of simpler methods like dot products (which struggle with different dimensions and only capture linear interactions) and cross products (which become computationally infeasible due to high dimensionality).

Datasets

  • Training Data: The system was trained using a dataset of memes scraped from Reddit. The visual pre-processor was trained to classify memes into one of 6200 subreddits. The text pre-processor was fine-tuned on the PTC Corpus (used in SemEval-2020 Task 11). The final multi-modal model was trained on the task organizers' released train set for SemEval-2021 Task 6.
  • Evaluation Data: The development set released by the task organizers was used for hyper-parameter optimization. The final evaluation was performed on the test set for SemEval-2021 Task 6, Subtask 3.

Key Insights and Ideas

  • Memes are inherently multi-modal: The paper strongly emphasizes that for many persuasion techniques, understanding both the text and the image is essential. Relying on only one modality is insufficient for accurate detection.
  • MCB Pooling is key for multi-modal fusion: The authors found that simply combining embeddings through weighted averaging was effective, but MCB Pooling provided a significant performance boost. This highlights the importance of sophisticated fusion techniques that can capture complex interactions between modalities.
  • Pre-training on related tasks is beneficial: Using pre-trained models for both visual (subreddit classification) and textual (BERT) components provided a strong foundation for the final task. The hypothesis that subreddit classification helps capture genre/emotion is supported by the performance gains.
  • Complex inferences remain a challenge: While MCB Pooling improves performance, the model still struggles with memes that require complex visual reasoning or inferring subtle ideas that go beyond direct textual or visual cues.

Key Results

The proposed multi-modal system using MCB Pooling achieved competitive results in SemEval-2021 Task 6, Subtask 3:

  • Macro F1-score: 0.263 (2nd best among participants)
  • Micro F1-score: 0.331 (14th best among participants)

These results significantly outperformed the baseline (Macro-F1: 0.051, Micro-F1: 0.070) and a simpler feature combination with weighted averaging (Macro-F1: 0.112, Micro-F1: 0.248). The authors noted that approximately 40% of the examples that were misclassified by weighted averaging but correctly predicted by MCB Pooling required combining information from both modalities.

Limitations and Future Work

  • Limitations: The paper acknowledges that the MCB model still fails when complex concepts or inferences are involved, particularly when subtle visual ideas need to be inferred in combination with the text. Jointly training visual and textual embeddings, rather than combining separate pre-trained embeddings, is suggested as a potential improvement.
  • Future Work: The authors plan to:
  • Incorporate additional semantic information, such as argumentation structures (e.g., logical fallacies) and emotional language features, to better capture the persuasive nature of memes.
  • Investigate approaches for jointly training visual and textual embeddings to create a more coherent and integrated understanding of memes. This is expected to be computationally more expensive and require larger datasets.

Summary updated

BibTeX
@inproceedings{singh2021lt,
  author = {Pranaydeep Singh and Els Lefever},
  title = {{LT3 at SemEval-2021 Task 6: Using Multi-Modal Compact Bilinear Pooling to Combine Visual and Textual Understanding in Memes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure