TL;DR
This paper presents BAMBAS's approach to SemEval-2024 Task 4, focusing on identifying persuasion techniques in internet memes using a lightweight, hierarchy-agnostic multilabel classification strategy. Their method leverages pre-trained language models for text embeddings and a binary relevance approach with oversampling to handle class imbalance, achieving competitive results without explicitly modeling the label hierarchy.
What Problem Does This Paper Address?
The paper tackles the challenge of automatically detecting persuasion techniques embedded within internet memes. Memes, while often humorous, are increasingly used to spread disinformation and manipulate public opinion, making their analysis crucial. This task, specifically SemEval-2024 Task 4, aims to build systems that can identify these techniques from meme text alone (Subtask 1) or from both text and images (Subtasks 2a and 2b). A key challenge is that the persuasion techniques are organized in a hierarchical structure, and the dataset exhibits significant class imbalance, making traditional multilabel classification difficult. The authors specifically investigate how effective a simpler approach, one that ignores this hierarchy, can be.
Methods
The BAMBAS team employed a three-component lightweight approach:
- Feature Extraction: They used pre-trained language models (PTLMs) to generate embeddings for the meme text. Specifically, they experimented with XLM-RoBERTa, BERTweet, and Bernice (a multilingual RoBERTa model trained on tweets). The [CLS] token embedding was used as the feature vector.
- Binary Relevance Strategy: Instead of trying to model the complex label hierarchy directly, they decomposed the multilabel problem into multiple independent binary classification problems. For each of the 20 persuasion technique labels, a separate binary classifier was trained.
- Handling Class Imbalance: Recognizing the significant imbalance in the dataset (average imbalance ratio of 47.38), they incorporated oversampling techniques. This involved using SMOTE (Synthetic Minority Over-sampling Technique) and a random oversampler to generate synthetic minority class examples for each binary classifier, aiming to balance the positive and negative instances.
Datasets
The primary dataset used was from SemEval-2024 Task 4, Subtask 1. This dataset consists of 8,500 English texts extracted from real internet memes, with 7,000 for training and the remainder split into validation and test sets. Each text can be assigned one or more labels from a set of 20 persuasion techniques, organized hierarchically. The paper also mentions that the task organizers made data available in four languages (English, Bulgarian, North Macedonian, and Arabic), but the BAMBAS team focused solely on the English challenge.
Key Insights and Ideas
- Hierarchy-Agnostic Approach Effectiveness: The paper demonstrates that a surprisingly effective multilabel classification system can be built without explicitly modeling the hierarchical relationships between labels. This challenges the assumption that complex hierarchical structures must be directly incorporated for good performance.
- Bernice's Strength for Memes: The Bernice PTLM, trained on a large corpus of informal tweets, proved to be the most effective feature extractor. This suggests that models trained on data with a similar style and domain (informal, social media text) are better suited for understanding meme content.
- Oversampling is Crucial: The significant improvement in results when employing oversampling techniques highlights the critical role of addressing class imbalance in this task. Without it, the performance of individual binary classifiers would likely be severely hampered.
- Binary Relevance Simplicity Pays Off: The binary relevance strategy, despite its inherent simplicity and disregard for label correlations, achieved competitive results, underscoring its robustness for this specific problem with a moderate number of labels.
Key Results
- The BAMBAS team's approach, using Bernice embeddings, binary relevance, and oversampling, exceeded the baseline by 21 percentage points in hierarchical F1 score on the English dataset.
- They ranked 23rd out of 31 teams in terms of hierarchical F1 score.
- However, they achieved a much higher rank of 11th in terms of hierarchical recall, suggesting their model was good at identifying positive instances, even if precision wasn't always top-tier.
- The final submission achieved a hierarchical F1 score of 0.5767 on the test set.
- Further experiments with multi-layer perceptrons (MLPs) showed that a fine-tuned approach (Bernice_class) outperformed their competition submission, achieving a test set H-F1 of 0.5840, indicating potential for further gains.
Limitations and Future Work
- Ignoring Hierarchy: While effective, the paper acknowledges that ignoring label correlations and hierarchical structures might limit performance compared to methods that explicitly model them.
- Oversampling for Memes: The authors suggest designing oversampling strategies specifically tailored to the unique characteristics of meme text, as general oversampling methods might not fully capture meme-specific nuances.
- Structured Output Classifiers: A promising future direction is to investigate structured output classifiers that can directly predict paths within the persuasion technique hierarchy, potentially leading to more nuanced and accurate predictions.
- Internal Nodes: The inclusion of internal nodes in the hierarchy (e.g., "Logos", "Ad Hominem") showed some improvement, particularly in recall, suggesting that incorporating some hierarchical information, even indirectly, can be beneficial.
Summary updated