TL;DR
This paper introduces "Themis," a novel multimodal model that combines Large Language Models (LLMs) and contrastive learning image encoders to detect persuasion techniques in memes. Themis achieved strong performance in the SemEval-2024 Task 4, demonstrating its effectiveness in identifying subtle forms of online manipulation.
What Problem Does This Paper Address?
The internet is awash with memes, which, due to their virality and immediate impact, are increasingly used to spread disinformation and manipulate public opinion. Identifying the persuasive techniques embedded within these multimodal content pieces is crucial for combating online manipulation. However, analyzing memes requires understanding both their visual and textual components, posing a significant challenge for traditional NLP or computer vision models. This paper tackles the problem of effectively classifying the presence of persuasion techniques in memes, especially across different languages.
Methods
Themis employs a modular architecture designed for multimodal meme analysis. Its core components include:
- Image Embedding Model: A contrastive learning image encoder (specifically, CLIP Base and CLIP Large) is used to extract meaningful visual features from meme images.
- Large Language Model (LLM): LLMs (specifically, TinyLlama, Phi-1.5, and Phi-2) are utilized to process the textual content of memes and align it with visual features in a shared embedding space.
- Image Embedding Projection: A projection layer maps image features into an LLM-compatible size.
- Token Merger Module: This novel component merges tokens from both image and text modalities, allowing the model to focus on salient aspects of the meme. It draws inspiration from patch merging in vision transformers but integrates both modalities.
- Token Averaging: Tokens from the LLM are averaged to create a single, semantically dense embedding for classification.
- Classification Head: A standard classification head takes the fused multimodal features to predict the presence of persuasion techniques.
To optimize training, the image embedding model and the LLM are frozen, while LoRA (Low-Rank Adaptation) weights are used to introduce long-range interactions and improve performance.
Datasets
The primary dataset used is from SemEval-2024 Task 4, Subtask 2b. This dataset consists of memes with corresponding text, labeled for the presence of persuasion techniques.
- Training Data: 1200 meme instances.
- Validation Data: 150 meme instances.
- Development Data: 300 meme instances.
- Test Data:
- English: 600 memes
- Bulgarian: 100 memes
- North Macedonian: 100 memes
- Arabic: 160 memes
The task organizers provided test sets in three "surprise" languages (Bulgarian, North Macedonian, and Arabic) to assess model robustness beyond English.
Key Insights and Ideas
- Multimodal Fusion is Key: The paper highlights that effectively combining visual and textual information is crucial for accurately detecting persuasion techniques in memes.
- LLMs Enhance Contextual Understanding: LLMs provide a powerful way to understand the nuances of meme text and its relationship with the image.
- Contrastive Learning for Visuals: Using pre-trained contrastive learning models like CLIP is effective for extracting rich visual representations from meme images.
- LoRA for Efficient Adaptation: LoRA weights offer a way to fine-tune large models efficiently, improving performance without retraining the entire model.
- Token Merging for Salience: The proposed token merger module is a novel approach to integrate multimodal information by focusing on the most relevant tokens.
- Cross-Lingual Transfer Challenges: While the model performs well on English, performance drops significantly on the "surprise" languages, indicating challenges in zero-shot cross-lingual transfer for this task.
Key Results
Themis achieved promising results in SemEval-2024 Task 4, Subtask 2b:
- English: Macro-F₁ score of 0.7986.
- Bulgarian: Macro-F₁ score of 0.6677.
- North Macedonian: Macro-F₁ score of 0.4791.
- Arabic: Macro-F₁ score of 0.5554.
The paper also reports an ablation study showing that larger image encoders (CLIP Large) generally yield better performance. The best Macro-F₁ score achieved during development was 0.83487, obtained with specific hyperparameter settings. Themis was ranked 5th for English and 2nd for Bulgarian on the official leaderboard.
Limitations and Future Work
- Cross-Lingual Performance: The model's performance on Bulgarian, North Macedonian, and Arabic is significantly lower than on English, highlighting a limitation in its cross-lingual generalization capabilities.
- Bias in Error Distribution: The confusion matrix shows a bias towards the propagandistic class, likely due to an imbalance in the training data.
- Sarcasm and Nuance: The model sometimes struggles with subtle forms of persuasion, such as sarcasm, which might require more specialized training data or techniques.
Future work could focus on improving cross-lingual transfer, addressing data imbalance, and developing more robust methods for detecting subtle persuasive techniques like sarcasm. The authors also suggest that further research into the interplay between text and image in memes is warranted.
Summary updated