IITK at SemEval-2024 Task 4: Hierarchical Embeddings for Detection of Persuasion Techniques in Memes

Shreenaga Chikoti, Shrey Mehta, Ashutosh Modi

Abstract

Memes are a potent medium for disinformation, blending textual and visual rhetoric to influence public opinion. This paper details the system developed by researchers at IIT Kanpur for SemEval-2024 Task 4, which focuses on the hierarchical multi-label classification of persuasion techniques in memes. The proposed approach leverages hyperbolic embeddings to capture the inherent tree-like structure of propaganda techniques and employs multi-modal fusion using CLIP and RoBERTa. Experimental results across English, Arabic, Bulgarian, and North Macedonian demonstrate the effectiveness of combining hierarchical awareness with semantic class definitions.

1. Introduction

The digital landscape is increasingly saturated with memes—multimodal artifacts that combine images and text to convey complex social and political messages. Their high shareability makes them ideal for spreading disinformation. Detecting the specific persuasion techniques used in memes is a significant challenge due to the subtle interplay between text and imagery and the hierarchical nature of the techniques themselves.

2. The Hierarchical Nature of Persuasion

Persuasion techniques are not independent categories; they exist within a structured taxonomy. For example, "Name Calling" and "Smears" are specific instances of "Ad Hominem" attacks. Traditional flat classification models often fail to respect these parent-child relationships, which is a central focus of this research.

Figure 5: Taxonomy of persuasion techniques used in the SemEval-2024 Task 4, illustrating the hierarchical levels from broad categories like Ethos, Pathos, and Logos down to specific techniques.

3. Methodology

The IITK team developed an ensemble model centered on three core innovations:

  • Hyperbolic Embeddings (HypEmo): Instead of standard Euclidean space, the model projects labels into a Poincaré ball (hyperbolic space). This is mathematically better suited for representing tree-like hierarchies because the space's volume grows exponentially with the radius, mirroring the growth of nodes in a taxonomy.
  • Class Definition Prediction (CDP): The model is trained to match meme content with the semantic definitions of persuasion techniques. This "dataset-agnostic" approach allows the model to rely on the semantic meaning of a technique rather than just statistical patterns.
  • Multi-modal Fusion: CLIP embeddings capture visual features, which are then fused with textual features from models like RoBERTa or DistilBERT to understand the full context of the meme.
Figure 4: Examples of memes from the dataset, showcasing different persuasive strategies and the integration of text and imagery.

4. Experimental Setup and Results

The system was evaluated on the SemEval-2024 Task 4 dataset, covering English and three other languages (Arabic, Bulgarian, North Macedonian). For non-English data, the team utilized OPUS-MT for translation.

Key performance metrics on the English test sets included:

  • Sub-task 1 (Text only): 0.60 Hierarchical F1
  • Sub-task 2 (Text + Visual): 0.67 Hierarchical F1
  • Sub-task 3 (Binary Detection): 0.48 Macro-F1

Notably, the team ranked 4th and 6th in the Arabic sub-tasks, highlighting the robustness of their multilingual strategy despite the training data being exclusively in English.

5. Discussion and Conclusion

The study confirms that visual cues are critical for identifying certain techniques like "Smears," while others rely more heavily on text. The integration of hierarchical information via hyperbolic embeddings and semantic definitions via CDP proved superior to baseline models. Future work will focus on more sophisticated multi-modal alignment and reducing the complexity of the ensemble architecture to improve inference efficiency.

Summary updated

BibTeX
@inproceedings{chikoti2024iitk,
  author = {Shreenaga Chikoti and Shrey Mehta and Ashutosh Modi},
  title = {{IITK at SemEval-2024 Task 4: Hierarchical Embeddings for Detection of Persuasion Techniques in Memes}},
  booktitle = {SemEval @ NAACL},
  year = {2024},
  doi = {10.18653/v1/2024.semeval-1.252},
}
Figure