TL;DR
This paper tackles the challenging task of automatically detecting persuasion techniques in multimodal memes across multiple languages. The authors propose an ensemble learning approach that leverages external knowledge from Google Vision to enrich image understanding, achieving competitive results on the SemEval-2024 Task 4.
What Problem Does This Paper Address?
Identifying persuasion techniques in online content, particularly memes, is crucial for combating misinformation and fostering critical media literacy. Memes are inherently multimodal, combining text and images, which makes their analysis complex. Furthermore, the spread of such content across different languages adds another layer of difficulty. Existing research often focuses on text-only analysis or single-language meme detection. This paper addresses the need for a robust, multilingual system capable of understanding the interplay between text and visuals to identify up to 22 distinct persuasion techniques, as defined by SemEval-2024 Task 4.
Methods
The core of the proposed system is a multimodal ensemble learning approach. For the primary task (Subtask2a: multilabel classification of persuasion techniques), the authors experimented with various combinations of NLP models (BERT, RoBERTa, mBERT, XLM-RBase) and vision models (VGG19, ResNet50, CLIP). They explored both early fusion (jointly training models and concatenating features) and late fusion (combining predictions from individual models).
A key novelty is the integration of external knowledge obtained from Google Vision's API. This API extracts "named visual entities" (objects, people, concepts) from images, providing a consistent representation across languages and reducing reliance on language-specific text features. This external data was processed by a BERT model and incorporated into the ensemble.
The authors also introduced a hierarchical F1 confidence threshold for Subtask2a. This mechanism allows for partial credit when a parent node of a technique is correctly identified, even if the specific technique is missed, and adjusts the confidence scores to better align with the task's hierarchical evaluation.
For Subtask2b (binary classification of any persuasion technique), the system also integrated visual and textual modalities, with a stronger emphasis on the visual component.
Datasets
The primary datasets used were provided for SemEval-2024 Task 4, including English training and development sets for both subtasks. Additionally, the task provided surprise languages (Arabic, North Macedonian, Bulgarian) for evaluation.
To augment the data, the authors employed:
- Direct translation of English training data into other languages using GPT-3.5.
- The Persuasion Techniques Corpus (PTC) was explored as additional training data, though its effectiveness was limited due to differing annotation conventions and domains.
- For post-evaluation experiments, they also utilized the Memotion dataset for additional non-offensive, non-propagandistic memes.
Key Insights and Ideas
- External visual entities are valuable: Integrating named visual entities from Google Vision proved beneficial, especially for multilingual scenarios, by providing a language-agnostic context for image interpretation.
- Ensembling pays off: Combining multiple NLP and vision models, particularly through late fusion, generally led to better performance than individual models.
- Hierarchy matters: The hierarchical F1 confidence threshold mechanism improved performance on Subtask2a by allowing for partial credit, reflecting the task's evaluation structure.
- Modality importance varies: While visual information was crucial for Subtask2b (binary classification), Subtask2a (multilabel classification) benefited from a more balanced integration of text and visual cues, with external visual entities acting as a concise representation.
- Language augmentation is tricky: While direct translation helped with multilingual generalization, it couldn't fully replicate the nuances of native language memes, especially concerning visual cultural context.
Key Results
- Subtask2a: The best-performing model for Subtask2a was the "Triad + BERT(ex)" model using late fusion, which achieved an F1 score of 0.483 on the English development set. The hierarchical F1 confidence threshold further improved this score. On the official test set, their model achieved competitive rankings across languages, with the best performance on North Macedonian.
- Subtask2b: The system achieved higher scores on Subtask2b compared to Subtask2a. Models like VGG19 + BERT showed strong performance on the English test set. The multilingual performance for Subtask2b was more varied, with a significant score reduction in North Macedonian.
Limitations and Future Work
- Cultural Nuances: The authors acknowledge that cultural elements within memes, particularly evident in the visually distinct North Macedonian memes, are under-researched and pose a challenge for cross-modal understanding.
- Augmentation Limitations: Direct translation for data augmentation cannot capture the visual modality's cultural context, potentially limiting generalization.
- Cross-Modal Understanding: While external visual entities helped, deeper cross-modal understanding or more sophisticated text analysis is still needed for complex persuasion techniques.
Future work could focus on:
- Developing methods for identifying named entities within the visual modality for richer knowledge graph integration.
- Generating higher-quality, multilingual data for persuasion techniques, especially for low-resource languages.
- Investigating the cultural aspects of meme production and consumption to improve multilingual zero-shot performance.
Summary updated