MemeMind at ArAIEval Shared Task: Generative Augmentation and Feature Fusion for Multimodal Propaganda Detection in Arabic Memes through Advanced Language and Vision Models

Uzair Shah, Md. Rafiul Biswas, Marco Agus, Mowafa Househ, Wajdi ZaghouaniACL 2024

TL;DR

This paper tackles the challenge of detecting propaganda in Arabic memes, a growing concern on social media. The authors propose a novel multimodal framework that leverages advanced language and vision models, combined with generative data augmentation (using ChatGPT4 and DALL-E-2) and a fusion layer, to significantly improve detection performance.

What Problem Does This Paper Address?

The proliferation of social media has led to an unprecedented spread of misinformation and propaganda. Memes, which combine text and images, have become a particularly effective medium for disseminating such content, especially in Arabic-speaking regions where research in this area is limited. Existing work has primarily focused on English-language memes, leaving a significant gap in understanding and combating propaganda in Arabic multimodal content. This paper addresses this gap by developing and evaluating a robust system for detecting propagandistic Arabic memes for the ArAIEval 2024 shared task.

Methods

The authors developed a three-task approach for multimodal propaganda detection in Arabic memes:

  • Task 2A (Text-based Propaganda Classification): They fine-tuned several state-of-the-art Arabic language models (e.g., AraBERT, MARBERT, CAMELBERT) and augmented the training data with synthetic text generated by ChatGPT4. This augmentation helped address class imbalance and improve performance.
  • Task 2B (Image-based Propaganda Classification): They fine-tuned popular image classification models like ResNet18, ConvNeXt-tiny, and EfficientNetV2. Synthetic images generated by DALL-E-2 were used to augment the dataset, particularly for the propaganda class, leading to performance gains.
  • Task 2C (Multimodal Propaganda Classification): This task involved fusing features extracted from both the text (using the best model from Task 2A, AraBERT) and image (using the best model from Task 2B, ConvNeXt-tiny). A fusion layer concatenated these features before feeding them into a binary classification layer. This approach aimed to leverage the complementary information from both modalities.

Key techniques include:

  • Generative Data Augmentation: Using ChatGPT4 for text and DALL-E-2 for images to create synthetic propaganda content, addressing class imbalance and improving model robustness.
  • Model Fine-tuning: Adapting pre-trained language and vision models to the specific task of Arabic meme propaganda detection.
  • Feature Fusion: Combining text and image features using a fusion layer for multimodal classification, aiming to capture richer semantic information.
  • Class Weighting: A standard technique to handle imbalanced datasets.

Datasets

The primary dataset used is from the ArAIEval 2024 shared task. It consists of pairs of text and images for training, validation, and testing. The dataset is noted to be imbalanced, with significantly more non-propaganda examples than propaganda examples in the training set. The authors also generated synthetic text and image data using ChatGPT4 and DALL-E-2, respectively, to augment the original dataset.

Key Insights and Ideas

  • Generative Augmentation is Crucial: Both text and image augmentation using large generative models (ChatGPT4, DALL-E-2) significantly boosted performance across all tasks, highlighting their effectiveness in addressing data scarcity and class imbalance in propaganda detection.
  • Multimodal Fusion Outperforms Unimodal: The multimodal approach (Task 2C), which fuses text and image features, achieved the highest performance, underscoring the importance of considering both modalities for accurate propaganda detection in memes.
  • Arabic Meme Propaganda is a Growing Challenge: The limited research on Arabic meme propaganda and the success of their approach emphasize the need for continued development of specialized models and datasets for this domain.
  • Fine-tuning Pre-trained Models is Effective: Leveraging state-of-the-art pre-trained language and vision models as a starting point significantly accelerates development and improves performance.

Key Results

The paper reports significant performance improvements due to data augmentation and the fusion approach.

  • Task 2A (Text): Augmenting with synthetic text led to notable improvements, with AraBERT achieving an impressive Macro F1-score of 0.80 during development.
  • Task 2B (Image): Augmenting propaganda images improved performance, with ConvNeXt-tiny achieving an F1-score of 0.736 with augmentation versus 0.729 without.
  • Task 2C (Multimodal): The fusion approach combining AraBERT and ConvNeXt-tiny yielded the highest Macro F1-score of 0.837 during development.
  • Leaderboard Performance (ArAIEval competition): The "Meme_mind" team achieved competitive results, securing a Macro F1-score of 0.7464 for Task 2A, 0.6634 for Task 2B, and 0.7972 for Task 2C.

Limitations and Future Work

  • Class Imbalance: While addressed through augmentation, class imbalance remains a challenge in propaganda detection datasets.
  • Generalizability: The models are trained on specific Arabic meme datasets; their generalizability to other Arabic social media content or different types of propaganda might be limited.
  • Computational Resources: Training and fine-tuning large language and vision models, especially with extensive data augmentation, requires significant computational resources.

Future work could focus on:

  • Developing more sophisticated fusion mechanisms.
  • Exploring cross-lingual transfer learning for propaganda detection.
  • Investigating adversarial attacks and defenses for multimodal propaganda detection.
  • Expanding the scope to include other forms of multimodal disinformation.

Summary updated

BibTeX
@inproceedings{shah2024mememind,
  author = {Uzair Shah and Md. Rafiul Biswas and Marco Agus and Mowafa Househ and Wajdi Zaghouani},
  title = {{MemeMind at ArAIEval Shared Task: Generative Augmentation and Feature Fusion for Multimodal Propaganda Detection in Arabic Memes through Advanced Language and Vision Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure