TL;DR
This paper tackles the challenge of automatically identifying persuasion techniques within meme text, a crucial aspect of understanding online disinformation. The authors developed an ensemble of language models enhanced with data augmentation through paraphrasing and custom threshold tuning, achieving strong performance across multiple languages, though with notable challenges in Arabic.
What Problem Does This Paper Address?
Memes are increasingly used in disinformation campaigns to subtly influence public opinion. Identifying the specific persuasion techniques employed within meme text is a complex task, especially given the hierarchical nature of these techniques and the multilingual nature of online content. SemEval-2024 Task 4, Subtask 1, specifically aimed to address this by requiring models to detect 20 distinct persuasion techniques within meme text, structured hierarchically. The challenge lies in building models that can generalize across languages and effectively handle the nuances of persuasive language in a meme context, often with limited training data for certain techniques.
Methods
The core of the CLaC team's approach involved an ensemble of three pre-trained language models: BERT, XLM-ROBERTa, and mBERT. These models were fine-tuned on a combined dataset. To address data scarcity, particularly for underrepresented persuasion techniques, they employed two key augmentation strategies:
- Dataset Augmentation with Prior Work: They incorporated data from the SemEval-2020 Task 11 dataset, which also focused on propaganda technique detection. This involved careful preprocessing to align the technique taxonomies between the two tasks.
- Paraphrase Generation: Leveraging ChatGPT-3.5 turbo, they generated paraphrases for existing training instances. This was done in stages, first creating datasets with 1 or 3 paraphrases per instance (Para-28k and Para-52k), and then focusing on generating 10 paraphrases for specific "benefited" techniques that showed improvement with paraphrasing, resulting in the Para-54k dataset.
- Per-Class Threshold Tuning: After generating predictions, they implemented custom classification thresholds for each persuasion technique. This was done to mitigate the impact of data imbalance and optimize performance on a per-class basis, using grid search on the validation set.
The final system was an ensemble of the fine-tuned models, using an unweighted average of their predictions. For testing on "surprise" languages (Bulgarian, North Macedonian, and Arabic), the input text was automatically translated into English, enabling zero-shot prediction.
Datasets
- SemEval-2024 Task 4 Training Data: Primarily English text from memes, with 7k samples.
- SemEval-2020 Task 11 Data (Comb-14k): Approximately 7k instances, used to augment the SemEval-2024 data.
- Paraphrased Datasets:
- Para-28k: Generated with n=1 paraphrase per instance.
- Para-52k: Generated with n=3 paraphrases per instance.
- Para-54k: Generated with 10 paraphrases for specific "benefited" techniques.
- Validation Set: 500 samples, used consistently for hyperparameter tuning and model selection.
- Development Set: 1k samples, used for final model evaluation before submission.
- Test Sets: English, Bulgarian, North Macedonian, and Arabic.
Key Insights and Ideas
- Paraphrasing as a Double-Edged Sword: While paraphrasing generally improved F1 scores, especially for certain techniques, it also introduced noise and, in some cases, decreased performance for others. This highlights the need for careful selection of augmentation strategies and techniques.
- Data Imbalance is a Persistent Challenge: Even with augmentation, the datasets remained imbalanced, necessitating techniques like per-class threshold tuning to achieve better overall performance.
- Ensembling for Robustness: Combining multiple language models proved effective, leveraging their diverse strengths and mitigating individual model weaknesses.
- Zero-Shot Translation is Viable but Imperfect: Translating test data into English for zero-shot prediction allowed the system to function on unseen languages, but performance varied, with Arabic being a particular challenge.
Key Results
- The ensemble model trained on the Para-52k dataset achieved the best hierarchical F1 score of 0.56 on the validation set.
- On the test sets, the system significantly outperformed the baseline in English, Bulgarian, and North Macedonian.
- For English, the system achieved a hierarchical F1 of 0.57827, compared to a baseline of 0.36865.
- Performance in Arabic was not significantly better than the baseline, with a score of 0.38070 versus a baseline of 0.35897.
- The authors ranked 22nd out of 33 participants for English, and within the top 12 for Bulgarian and North Macedonian.
Limitations and Future Work
- Arabic Performance: The lack of significant improvement in Arabic is a key limitation. The authors suggest this might be due to the data augmentation and threshold fine-tuning introducing noise or exacerbating dataset imbalance in that specific language.
- Dataset Imbalance: Despite efforts, the datasets remained imbalanced, impacting performance on underrepresented classes.
- Noise from Augmentation: The paraphrasing strategy, while beneficial, also introduced noise, leading to decreased performance in some cases.
Future work could focus on more sophisticated methods for handling data imbalance, exploring language-specific fine-tuning for the surprise languages, and developing more targeted data augmentation techniques that minimize noise introduction. Further analysis into why Arabic performance lagged would also be valuable.
Summary updated