TL;DR
This paper tackles the challenge of automatically classifying propaganda techniques in multimodal content (text and images), a crucial task in combating misinformation. The authors propose a transformer-based architecture that effectively fuses visual and textual information, achieving state-of-the-art results and winning first place in the SemEval-2021 Task 6-c competition.
What Problem Does This Paper Address?
In today's digital landscape, propaganda is a pervasive issue, often disseminated through multimodal content like memes. These memes combine text and images to influence audiences, making their detection a complex problem. While previous research has focused on analyzing text alone, this paper addresses the need for a system that can understand and classify propaganda techniques by considering both the textual and visual elements of such content. Specifically, SemEval-2021 Task 6 aimed to identify which of 22 propaganda techniques are present in multimodal meme data, a task that requires sophisticated methods for integrating information from different modalities.
Methods
The core of the proposed solution lies in leveraging transformer architectures for multimodal fusion. The authors explore two main approaches:
- Fine-tuning Text Transformers with Extended Visual Features: This method starts with a pre-trained text transformer (like DeBERTa) and augments its input with visual features. These visual features are extracted using two strategies:
- Grid Features: Convolutional Neural Networks (CNNs) like ResNet are used to extract features from uniformly divided image regions.
- Salient Region Features: Object detection models (e.g., Faster R-CNN) are employed to identify important regions in the image, and their features are then used.
- Fine-tuning Multimodal Pre-trained Transformers: This approach utilizes models specifically designed for joint vision-language understanding. The authors highlight ERNIE-ViL, a two-stream transformer model pre-trained on large-scale image-caption data, as a particularly effective choice. ERNIE-ViL models visual and textual features independently before fusing them.
A key challenge identified is the highly imbalanced nature of the dataset (many more negative samples than positive ones for each propaganda technique). To address this, the authors experiment with Focal Loss as an alternative to standard cross-entropy loss, aiming to give more weight to misclassified samples, especially those from minority classes.
Datasets
The paper uses data from SemEval-2021 Task 6. While the exact size of the training, development, and test sets are mentioned in terms of example counts and class distributions, no new benchmark dataset was introduced by the authors. The dataset is characterized by its multimodal nature (text and images) and the presence of 22 propaganda technique classes, with a significant imbalance in the distribution of positive and negative samples across these classes.
Key Insights and Ideas
- Multimodal Fusion is Crucial: The results strongly indicate that incorporating visual information significantly improves propaganda classification performance compared to relying solely on text.
- Pre-trained Multimodal Models Shine: Fine-tuning a pre-trained multimodal transformer like ERNIE-ViL yields superior results compared to augmenting a text-only transformer with visual features. This suggests that models pre-trained on joint vision-language tasks have learned more effective cross-modal representations.
- Object Detection Features are Promising: Salient region features extracted by object detectors (like BUTD) tend to perform better than grid-based features from CNNs, likely because they focus on semantically relevant parts of the image.
- Focal Loss Tackles Imbalance: The imbalanced nature of the propaganda classification task is a significant hurdle. Focal loss proves to be more effective than cross-entropy loss in improving performance, particularly the F1 score, by better handling the minority classes.
- More Regions Can Be Better, Up to a Point: For object detection-based features, increasing the number of detected regions generally improves performance, but there's a point of diminishing returns (e.g., 100 boxes were worse than 0-36) where irrelevant or low-confidence regions might introduce noise.
Key Results
The paper reports achieving first place in the SemEval-2021 Task 6-c multimodal classification task.
- The best single model, ERNIE-ViL, achieved a test F1 score of 57.14.
- An ensemble of their top three models achieved a test F1 score of 58.11, securing their top ranking.
- Using Focal Loss improved the F1 score from 64.35 (cross-entropy) to 68.56 when compared to the baseline with cross-entropy loss.
- Visual features from object detectors (BUTD) showed a slight improvement over ResNet50 features, achieving a test F1 of 56.21 compared to 55.96.
Limitations and Future Work
- Dataset Size: The authors acknowledge that the training dataset is relatively small, which motivates their use of ensemble methods to maximize performance and diversity.
- Object Region Count: They observed that while more object regions can be beneficial, an excessive number can degrade performance, suggesting that careful selection or filtering of regions is important.
- Future Work: The paper implicitly suggests that further exploration of multimodal transformer architectures and more sophisticated feature fusion techniques could lead to even better performance. The success of ERNIE-ViL also points towards the continued importance of large-scale pre-training for vision-language tasks.
Summary updated