TL;DR
This paper tackles the challenge of classifying persuasion techniques in memes, a complex task due to the hierarchical and multi-label nature of the categories. The authors developed a novel hierarchical classification head for Large Language Models (LLMs) that effectively captures label dependencies, leading to top performance across multiple languages in the SemEval-2024 Task 4.
What Problem Does This Paper Address?
Memes, while often humorous, are increasingly recognized as potent tools for persuasion and manipulation. Identifying the specific techniques used in memes is crucial for understanding and combating their influence. This task is particularly challenging because persuasion techniques form a hierarchy (e.g., "Ethos" is a broad category with sub-techniques like "Appeal to Authority") and a single meme can employ multiple techniques simultaneously (multi-label classification). Existing methods often struggle to effectively model these hierarchical relationships and the inherent imbalance in label frequencies within datasets.
Methods
The core of the proposed system is a custom hierarchical classification head designed to be appended to pre-trained Large Language Models (LLMs). This head consists of multiple fully connected layers, where each layer builds upon the decisions of the previous one, mirroring the hierarchical structure of the labels. This allows for information flow from broader categories to more specific ones.
Key methodological aspects include:
- Hierarchical Architecture: The classification head explicitly models the label hierarchy, with layers corresponding to different levels of the persuasion technique taxonomy.
- LLM Integration: The system leverages powerful pre-trained LLMs (like BERT, RoBERTa, and LLaMA 2) as feature extractors, benefiting from their robust understanding of language.
- Hyperparameter Optimization: Extensive grid searches were conducted to find optimal parameters for various models, including learning rates, pre-processing styles (cleaned text vs. all lowercase), and the use of weighted loss functions.
- Multilingual Handling: For non-English datasets, a translation step using GPT-4 was employed to convert meme text into English, enabling the use of the English-trained model.
- Ensemble Approach: To further boost performance and robustness, an ensemble of four independently trained models was used, with majority voting determining the final prediction.
Datasets
The primary dataset used was the SemEval-2024 Task 4 Subtask 1 dataset, which includes:
- English: Training, validation, and test sets.
- Bulgarian, North Macedonian, and Arabic: Test sets were also provided.
The paper notes that the distribution of labels within these datasets is highly uneven, with some techniques appearing much more frequently than others.
Key Insights and Ideas
- Hierarchy Matters: Explicitly modeling the hierarchical structure of labels in the classification head significantly improves performance, outperforming simpler multi-label approaches.
- LLMs are Powerful, but Custom Heads are Key: While LLMs provide strong embeddings, a custom head tailored to the hierarchical nature of the task is crucial for achieving state-of-the-art results.
- Weighted Loss Can Hurt: Surprisingly, weighting the loss function to address label imbalance did not improve performance and, in some cases, degraded it. This might be due to the extreme imbalance in the dataset.
- Global Thresholds are Better: Using a single, global classification threshold across all labels proved more effective than optimizing individual thresholds for each label.
- Translation is Effective: Translating non-English meme text to English using a sophisticated prompt-based approach (GPT-4) allowed the system to perform well on multilingual datasets without needing separate models for each language.
Key Results
The proposed system achieved remarkable success:
- Third place on the English test dataset.
- First place on the Bulgarian, North Macedonian, and Arabic test datasets.
- The custom classification head improved results by up to eleven percent points compared to baseline LLM setups.
- The ensemble model achieved a hierarchical F1-score of 0.690 on the dev set, outperforming individual models.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Label Imbalance: The significant imbalance in label frequencies remains a challenge, particularly for rare persuasion techniques, leading to zero F1-scores for some.
- Parent Node Selection: The current method of including all ancestors of a predicted leaf might lead to compounded errors. Future work could refine this by discarding parent nodes if their own confidence is low.
- Pre-training Data: Utilizing similar data sources for pre-training LLMs could further enhance performance.
- Larger LLMs: Exploring even larger LLMs (e.g., LLaMA 2 70b) could yield better results.
- Hyperparameter Tuning: Further extensive hyperparameter tuning might uncover even better model configurations.
- Feature Stacking: Investigating feature stacking, where features from different models are combined, could lead to more powerful models.
Summary updated