Abstract
Memes are a potent medium for disinformation, blending textual and visual rhetoric to influence public opinion. This paper details the system developed by researchers at IIT Kanpur for SemEval-2024 Task 4, which focuses on the hierarchical multi-label classification of persuasion techniques in memes. The proposed approach leverages hyperbolic embeddings to capture the inherent tree-like structure of propaganda techniques and employs multi-modal fusion using CLIP and RoBERTa. Experimental results across English, Arabic, Bulgarian, and North Macedonian demonstrate the effectiveness of combining hierarchical awareness with semantic class definitions.
1. Introduction
The digital landscape is increasingly saturated with memes—multimodal artifacts that combine images and text to convey complex social and political messages. Their high shareability makes them ideal for spreading disinformation. Detecting the specific persuasion techniques used in memes is a significant challenge due to the subtle interplay between text and imagery and the hierarchical nature of the techniques themselves.
2. The Hierarchical Nature of Persuasion
Persuasion techniques are not independent categories; they exist within a structured taxonomy. For example, "Name Calling" and "Smears" are specific instances of "Ad Hominem" attacks. Traditional flat classification models often fail to respect these parent-child relationships, which is a central focus of this research.
3. Methodology
The IITK team developed an ensemble model centered on three core innovations:
- Hyperbolic Embeddings (HypEmo): Instead of standard Euclidean space, the model projects labels into a Poincaré ball (hyperbolic space). This is mathematically better suited for representing tree-like hierarchies because the space's volume grows exponentially with the radius, mirroring the growth of nodes in a taxonomy.
- Class Definition Prediction (CDP): The model is trained to match meme content with the semantic definitions of persuasion techniques. This "dataset-agnostic" approach allows the model to rely on the semantic meaning of a technique rather than just statistical patterns.
- Multi-modal Fusion: CLIP embeddings capture visual features, which are then fused with textual features from models like RoBERTa or DistilBERT to understand the full context of the meme.
4. Experimental Setup and Results
The system was evaluated on the SemEval-2024 Task 4 dataset, covering English and three other languages (Arabic, Bulgarian, North Macedonian). For non-English data, the team utilized OPUS-MT for translation.
Key performance metrics on the English test sets included:
- Sub-task 1 (Text only): 0.60 Hierarchical F1
- Sub-task 2 (Text + Visual): 0.67 Hierarchical F1
- Sub-task 3 (Binary Detection): 0.48 Macro-F1
Notably, the team ranked 4th and 6th in the Arabic sub-tasks, highlighting the robustness of their multilingual strategy despite the training data being exclusively in English.
5. Discussion and Conclusion
The study confirms that visual cues are critical for identifying certain techniques like "Smears," while others rely more heavily on text. The integration of hierarchical information via hyperbolic embeddings and semantic definitions via CDP proved superior to baseline models. Future work will focus on more sophisticated multi-modal alignment and reducing the complexity of the ensemble architecture to improve inference efficiency.
Summary updated