CLTL at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Transformer Models

Yeshan Wang, Ilia MarkovACL 2024

TL;DR

This paper presents the CLTL team's approach to the ArAIEval 2024 Shared Task, focusing on classifying propagandistic Arabic memes. They achieved strong results by leveraging state-of-the-art transformer models for both text and image modalities, and a Multilayer Perceptron (MLP) for multimodal fusion, securing 2nd, 1st, and 3rd place in the respective subtasks.

What Problem Does This Paper Address?

In the digital age, memes have become a powerful tool for spreading propaganda and misinformation. Detecting these propagandistic memes is challenging due to their multimodal nature, combining text and images, and the use of subtle persuasive techniques like logical fallacies and emotional appeals. While previous research has focused on either textual or visual propaganda detection, the ArAIEval 2024 Shared Task specifically addresses the multimodal classification of Arabic memes, a novel and crucial area for understanding online influence.

Methods

The CLTL team employed a multi-pronged strategy for the three subtasks:

  • Subtask 2A (Textual Modality): They experimented with several state-of-the-art Arabic transformer language models, including MARBERT, CAMELBERT-MIX-SA, and GigaBERT. These models were fine-tuned on the provided Arabic meme text.
  • Subtask 2B (Visual Modality): For image classification, they utilized two advanced vision models: EVA, a vision-centric foundation model, and CAFormer, a model designed for efficient computation and capturing long-range dependencies.
  • Subtask 2C (Multimodal): To combine text and image information, they adopted a Multilayer Perceptron (MLP) fusion module. This module concatenates the embeddings extracted by the unimodal models (textual from language models and visual from vision models) and feeds them into a prediction layer for final classification. This architecture is illustrated in Figure 1.

A key aspect of their methodology was the decision not to apply extensive text preprocessing, relying instead on the pre-trained transformer models to handle input representations directly. All models were trained with consistent hyperparameters, including a base learning rate of 1e-4 and a maximum of 10 epochs.

Datasets

The primary dataset used for all three subtasks was provided by the ArAIEval 2024 Shared Task. This dataset consists of Arabic memes sourced from platforms like Facebook, Twitter, Instagram, and Pinterest. Each meme is labeled as either "propagandistic" or "non-propagandistic" and includes both the original image and its corresponding textual content. The paper notes that the dataset is imbalanced, with the "propagandistic" class being the minority class across training, development, and test sets.

Key Insights and Ideas

  • Textual Modality Dominance: The results suggest that textual information is more indicative of propaganda in memes than visual information alone. The best-performing text-based model (MARBERT) achieved a higher F1 score on its own than the best vision-based model (EVA).
  • Multimodal Fusion Benefits: Combining text and image modalities through MLP fusion led to a further performance improvement, highlighting the synergistic power of multimodal analysis for propaganda detection.
  • Model Selection Matters: Different transformer models showed varying strengths. MARBERT excelled in textual analysis, while EVA outperformed CAFormer in visual analysis. For multimodal fusion, GigaBERT combined with EVA yielded the best results.
  • Minimal Preprocessing Effectiveness: The approach of feeding raw text representations into transformer models, without extensive preprocessing, proved effective, simplifying the pipeline and leveraging the models' inherent capabilities.

Key Results

The CLTL system achieved the following macro-averaged F1 scores:

  • Subtask 2A (Text): 77.96% (2nd place)
  • Subtask 2B (Vision): 71.04% (1st place)
  • Subtask 2C (Multimodal): 79.80% (3rd place)

Specifically, MARBERT was the top-performing language model for subtask 2A on the development set, achieving 80.48 F1. For subtask 2B, EVA achieved 76.89 F1 on the development set. In the multimodal subtask 2C, the GigaBERT + EVA combination showed the highest performance on the development set (82.60 F1), and their submitted model achieved 79.80 F1 on the test set. Confusion matrices (Figures 2, 3, and 4) illustrate the performance of the submitted models, showing that the multimodal model for subtask 2C had the best performance with the lowest false positive and false negative rates.

Limitations and Future Work

  • Dataset Imbalance: The paper acknowledges the imbalanced nature of the dataset, which can pose challenges for model training and evaluation.
  • Vision Model Performance: The vision model for subtask 2B performed the worst among the three subtasks, indicating potential areas for improvement in visual propaganda detection.
  • Future Directions: While not explicitly detailed, the success of multimodal fusion suggests that further exploration of advanced fusion techniques and more robust vision models could lead to even better performance in future propaganda detection tasks. The paper's success with minimal preprocessing also opens avenues for exploring how much benefit can be gained from sophisticated text cleaning and feature engineering in conjunction with powerful transformer models.

Summary updated

BibTeX
@inproceedings{wang2024cltl,
  author = {Yeshan Wang and Ilia Markov},
  title = {{CLTL at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Transformer Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure