TL;DR
This paper presents a multilingual ensemble learning system for detecting persuasion techniques in meme texts, achieving competitive results in the SemEval-2024 Task 4. The approach leverages a hierarchical structure of persuasion techniques and a novel bagging strategy to address data imbalances, demonstrating its effectiveness across multiple languages.
What Problem Does This Paper Address?
Memes, while often humorous, can be powerful tools for spreading persuasive messages, sometimes even misleading ones. Identifying these persuasion techniques within meme texts is a complex task because:
- Multilabel Nature: A single meme can employ multiple persuasion techniques simultaneously.
- Hierarchical Structure: Persuasion techniques are not isolated; they often exist in a hierarchical relationship (e.g., a broad category like "Ethos" might contain sub-techniques like "Appeal to Authority").
- Data Imbalance: Real-world datasets for persuasion detection often suffer from significant imbalances, with some techniques appearing far more frequently than others, and imbalances between positive and negative examples for each technique.
- Multilingual Challenge: Persuasion techniques can manifest differently across languages, making a multilingual approach crucial.
This paper tackles these challenges by developing a system to automatically detect persuasion techniques in meme texts across various languages, specifically addressing the SemEval-2024 Task 4.
Methods
The core of the proposed system is a bagging-based ensemble learning approach combined with a soft voting strategy. Here's a breakdown:
- Ensemble Learning: Instead of relying on a single model, the system trains multiple "base learners" (in this case, XLM-RoBERTa-large models) and combines their predictions. This is a common technique to improve robustness and accuracy.
- Bagging Strategy: The training data is divided into subsets. Each base learner is trained on one of these subsets. This helps in mitigating overfitting and handling data imbalances.
- Hierarchical Grouping: To leverage the hierarchical nature of persuasion techniques, the authors group them into seven subsets based on their structure. This grouping is further refined to create ten distinct subsets, aiming to balance the data distribution for training.
- Soft Voting: Base learners predict the probability of each technique being present. A "weight model" then assigns weights to the predictions of each base learner. These weighted predictions are averaged (soft voting) to produce the final prediction. This allows the system to dynamically adjust the influence of each base learner.
- Data Augmentation (Implicit): The system utilizes data from both SemEval-2024 (memes) and SemEval-2023 (news articles) to create a more comprehensive training set, especially for lower-resource techniques.
What's novel here is the specific way they group techniques to create balanced subsets for training base learners and the use of a dedicated weight model to learn optimal weights for the soft voting ensemble, aiming to better exploit the hierarchical structure and address data imbalances.
Datasets
The system was trained and evaluated using data from two main sources:
- SemEval-2024 Task 4 Subtask 1: This corpus contains English meme texts annotated with 20 persuasion techniques. This was the primary dataset for the task.
- SemEval-2023 Task 3 Subtask 3: This corpus includes news articles in six languages (English, German, French, Russian, Polish, and Italian) annotated with 23 persuasion techniques. This dataset was used to supplement the SemEval-2024 data, particularly for creating training subsets for lower-resource techniques.
Key Insights and Ideas
- Hierarchical Grouping is Key: The authors' strategy of grouping techniques based on their hierarchical structure and then further refining these groups to create balanced training subsets was crucial for handling the complex nature of persuasion techniques and data imbalances.
- Ensemble Power: The ensemble approach, particularly with soft voting, proved effective in combining the strengths of multiple models, allowing them to generalize better across different techniques and languages.
- Weight Model's Potential (and Pitfalls): The idea of a dedicated weight model to learn optimal ensemble weights is promising. However, the paper notes that its performance was not significantly better than a baseline, suggesting room for improvement in how these weights are learned.
- Zero-Shot Performance: The system showed competitive results in languages it was not explicitly trained on (zero-shot setting), highlighting the robustness of the multilingual XLM-RoBERTa base learners.
Key Results
The system achieved competitive results in the SemEval-2024 Task 4 Subtask 1:
- Official Ranking:
- Second position in Bulgarian and North Macedonian.
- Fifth position in Arabic.
- Eleventh position in English.
- Comparison to Baseline: The ensemble approach with the weight model performed similarly to a baseline (where weights were set to 1) in English, Bulgarian, and North Macedonian. However, the baseline slightly outperformed the proposed approach in Arabic. This suggests the weight model's effectiveness varied across languages.
- Error Analysis: The analysis revealed that certain techniques like "Loaded Language" and "Name calling/Labeling" were consistently mispredicted. The authors attribute this partly to the chosen classification threshold and the insufficient accuracy of the weight model.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Weight Model Accuracy: The accuracy of the weight model was not as high as desired, and the gap between weights assigned to target subsets and others was not always significant. Future work could explore more sophisticated training techniques for the weight model.
- Data Imbalance Persistence: Despite efforts, data imbalance, particularly between positive and negative instances for some techniques, remains a challenge.
- Threshold Sensitivity: The fixed classification threshold (0.22) might be too small for techniques with rich training instances, leading to mispredictions.
- Technique Grouping Refinement: Future work could focus on developing more sophisticated technique grouping strategies that better account for the positive/negative instance imbalance within techniques.
- Code Availability: The authors have made their code publicly available, encouraging further research and development in this area.
Summary updated