SuteAlbastre at SemEval-2024 Task 4: Predicting Propaganda Techniques in Multilingual Memes using Joint Text and Vision Transformers

Ion Anghelina, Gabriel Buță, Alexandru EnacheNAACL 2024

TL;DR

This paper presents a multimodal approach for detecting propaganda techniques in memes, combining text and image information. The team achieved strong results, particularly on tasks involving both text and images, by leveraging pre-trained Vision and Text Transformers.

What Problem Does This Paper Address?

In today's digital landscape, social media plays a significant role in shaping public opinion. Memes, a popular form of online content, can be easily weaponized to spread political propaganda. Automatically detecting these subtle forms of manipulation is a challenging task, especially given the multimodal nature of memes (combining text and images) and the need to support multiple languages. SemEval-2024 Task 4 specifically aims to address this by tasking participants with identifying various persuasion techniques used in memes.

Methods

The core of the proposed solution is a multimodal neural network architecture that utilizes both Textual and Vision Transformers. For the visual component, they experimented with EfficientNet and Vision Transformer (ViT) models. On the textual side, they employed BERT and GPT-2. The system was designed to handle different subtasks: Subtask 1 focuses on text-only memes, while Subtasks 2a and 2b incorporate image information.

For multilingual support, especially for languages where no training data was provided, they employed a One-Shot Learning approach. This involved training on English data and then adapting the model for other languages. A key aspect of their approach for Subtasks 2a and 2b was the joint processing of text and image features, which were then fed into a fully connected neural network for classification. For Subtask 1, only text features were used.

Datasets

The primary dataset used for this task is the SemEval-2024 Task 4 dataset, which includes memes with English captions. For multilingual tasks (Arabic, Bulgarian, North Macedonian), only English training data was available, necessitating a One-Shot Learning approach. The paper mentions using pre-trained models that were themselves fine-tuned on datasets like HatefulMemes and HarMemes.

Key Insights and Ideas

  • Multimodal Fusion is Key: The paper highlights that combining text and image information is crucial for effectively detecting propaganda in memes, especially for Subtasks 2a and 2b, where they achieved their best results.
  • Transformer Power: The use of pre-trained Vision Transformers (ViT) and Text Transformers (BERT, GPT-2) proved highly effective, allowing the model to capture complex features from both modalities.
  • One-Shot Learning for Multilingualism: The strategy of training on English data and adapting for other languages was a practical solution for tackling multilingual tasks with limited or no specific training data.
  • Adaptive Thresholding for Regression Tasks: For Subtask 1 and 2a, which were framed as regression tasks, experimenting with adaptive thresholding strategies was important for optimizing performance.

Key Results

The team achieved strong performance, ranking second in Subtask 2a and Subtask 2b across multiple languages, with F1-scores exceeding 0.8 in the binary setup for Subtask 2b. They outperformed the competition baseline for Subtasks 2a and 2b in all provided languages. However, their performance on Subtask 1, particularly for languages without training samples, was less impressive, with an average F1-score around 0.2.

Limitations and Future Work

The authors acknowledge that their model struggled with Subtask 1, especially for languages lacking training data. They also mention that their adaptive thresholding approach was tested after the competition, suggesting that further exploration of this area could yield improvements. For future work, they propose exploring decision-making on a Hierarchical DAG to better utilize the relationships between different propaganda technique labels. They also suggest the use of additional datasets in various languages to further enhance the model's capabilities.

Summary updated

BibTeX
@inproceedings{anghelina2024sutealbastre,
  author = {Ion Anghelina and Gabriel Buță and Alexandru Enache},
  title = {{SuteAlbastre at SemEval-2024 Task 4: Predicting Propaganda Techniques in Multilingual Memes using Joint Text and Vision Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure