TL;DR
This paper introduces DVTT, a novel multimodal transformer architecture designed to classify persuasion techniques in memes by jointly processing text and images. DVTT achieves competitive results on SemEval-2021 Task 6, outperforming a baseline transformer model and demonstrating the effectiveness of its cross-modal attention mechanism.
What Problem Does This Paper Address?
In today's digital landscape, social media platforms are rife with disinformation campaigns that often leverage the power of memes. Memes, being concise and visually engaging, can effectively spread ideas and influence public opinion through various rhetorical and psychological techniques. Automatically detecting these persuasion techniques is crucial for managing the spread of misinformation. This paper tackles the challenge of identifying these techniques in memes, specifically addressing SemEval-2021 Task 6, which involves classifying memes based on their textual content alone (Subtask 1) and their combined text and image content (Subtask 3).
Methods
The core of the proposed solution is the Double Visual Textual Transformer (DVTT) architecture. DVTT builds upon the successful transformer model, extending it to handle multimodal inputs. Unlike simpler approaches that might process text and images separately, DVTT employs two interconnected transformer networks: one for text and one for images. The key innovation lies in how these two networks are conditioned on each other. This allows the model to reason jointly across modalities, where one modality can enrich the understanding derived from the other.
Specifically, DVTT uses two full transformer networks. One network processes the text, and the other processes the images. Crucially, the output of each transformer is used to condition the other, enabling a deeper, cross-modal understanding. The final predictions for each persuasion technique are obtained by averaging the probabilities from both transformers. The entire network is trained end-to-end using a binary cross-entropy loss.
As a baseline, the authors also propose the Visual-Textual Transformer Encoder (VTTE), which uses a transformer encoder to process pre-extracted visual and textual features. For subtask 1 (text-only), VTTE is used without visual input.
Datasets
The study utilizes the dataset provided by the SemEval-2021 Task 6 organizers. This dataset consists of:
- 687 memes for training
- 63 memes for validation (development set)
- 200 memes for testing
The annotations for the memes include lists of persuasion techniques. Subtask 1 involves 20 possible techniques, while Subtask 3 includes 22. The authors note that they were allowed to use the development set for training, resulting in a total of 750 annotated memes for training and validation.
Key Insights and Ideas
- Cross-modal conditioning is key: The DVTT model's success highlights the importance of explicitly modeling the interaction between text and images in memes. Simply processing them independently and then combining results is less effective than allowing them to influence each other's reasoning.
- Transformer's versatility: The paper demonstrates the transformer architecture's adaptability beyond its original NLP applications, showing its effectiveness in jointly processing visual and textual data for complex classification tasks.
- Fine-tuning for domain adaptation: Fine-tuning pre-trained feature extractors (BERT for text and ResNet50 for images) significantly improves performance, suggesting that adapting general-purpose models to the specific domain of memes is beneficial.
- Contextual information is challenging: The authors acknowledge that for some complex persuasion techniques, external knowledge or deeper common-sense reasoning might be necessary, indicating a limitation of purely data-driven approaches for certain nuanced labels.
Key Results
The DVTT model achieved competitive results on the SemEval-2021 Task 6 leaderboard:
- Subtask 3 (Images + Text): The DVTT model reached 4th place on the public leaderboard. The best performing DVTT variant, with fine-tuned BERT and ResNet50 modules, achieved a Macro-F₁ score of 0.341 and a Micro-F₁ score of 0.592.
- Subtask 1 (Text Only): Using the VTTE baseline model, the team secured 5th place on the public leaderboard. Fine-tuning the BERT model for this subtask also yielded improvements.
Ablation studies showed that the DVTT model generally outperformed the VTTE baseline. The choice of four encoder and decoder layers for DVTT seemed to offer the best compromise between metrics.
Limitations and Future Work
The authors identify several limitations and suggest future research directions:
- Visual Feature Extraction: Improving the visual feature extraction pipeline is a key area for future work. This could involve incorporating techniques like face expression detection and classification, or developing ad-hoc object detectors specifically trained for meme content.
- External Knowledge Integration: For more complex persuasion techniques that rely on common sense and historical facts, integrating external knowledge bases could enhance the model's reasoning capabilities.
- Understanding Attention Mechanisms: Further investigation into the transformer's attention mechanisms could provide deeper insights into how the model reasons about memes and their persuasive elements.
Summary updated