TL;DR
This paper explores a classical machine learning approach for identifying propagandistic memes across multiple languages. By training a robust English baseline and then leveraging machine translation to adapt it to Bulgarian, North Macedonian, and Arabic, the authors demonstrate the potential of simpler models while highlighting the significant challenges introduced by translation inaccuracies and language-specific nuances.
What Problem Does This Paper Address?
The proliferation of propaganda on social media, particularly through memes, poses a growing threat to public discourse and democratic processes. Identifying these propagandistic memes, especially across diverse linguistic landscapes, is a critical but challenging task for Natural Language Processing (NLP). While state-of-the-art Large Language Models (LLMs) are often the go-to for such tasks, this paper investigates whether a more traditional, classical machine learning approach, augmented by machine translation, can achieve competitive results for multilingual meme propaganda detection. The specific challenge is to build a system that can effectively classify memes as "propagandistic" or "non_propagandistic" across English, Bulgarian, North Macedonian, and Arabic.
Methods
The core of the methodology revolves around building a strong English baseline using a classical machine learning approach. The authors trained a SpaCy v3 model on a substantial English dataset. This baseline was then adapted for other languages by translating the non-English memes into English using Google Translate. The same trained English model was then applied to these translated memes. This approach aims to leverage the linguistic understanding gained from the English data for multilingual classification without needing to train separate models for each language from scratch.
Datasets
The study utilized official datasets from SemEval-2024 Task 4, specifically subtasks 1, 2a, and 2b. Additionally, data from Kaggle was incorporated to augment the training set. The combined training data aimed for approximately 11,001 "propagandistic" and 11,001 "non_propagandistic" memes, totaling around 22,000 for the English baseline training. For the multilingual aspect, unlabeled test data in Bulgarian, North Macedonian, and Arabic were used.
Key Insights and Ideas
- Classical ML Still Has a Place: The paper demonstrates that even with the rise of LLMs, classical machine learning models, when trained on well-prepared data, can achieve surprisingly good performance on complex tasks like propaganda detection.
- Machine Translation is a Double-Edged Sword: While machine translation offers a practical way to extend a single-language model to multiple languages, its accuracy directly impacts performance. Subtle shifts in meaning or loss of nuance can significantly degrade classification scores.
- Language and Topic Nuances Matter: The performance drop in non-English languages isn't solely due to translation. Differences in prevalent topics and cultural contexts (e.g., specific political disputes between countries) can also hinder a model trained on a different linguistic and cultural landscape.
- Data Augmentation is Crucial: Combining multiple datasets and carefully preparing the training data (e.g., lowercasing, removing usernames) was key to achieving a strong English baseline.
Key Results
The English baseline achieved a respectable F1 macro score of 0.71353 and an F1 micro score of 0.79000, ranking 13th out of 20 participants. However, the performance dropped significantly for other languages:
- Bulgarian: F1 macro 0.32670, F1 micro 0.33000 (ranked 14/14)
- North Macedonian: F1 macro 0.38942, F1 micro 0.46000 (ranked 13/14)
- Arabic: F1 macro 0.52825, F1 micro 0.54375 (ranked 9/14)
Arabic showed the best performance among the non-English languages, which the authors attribute to potentially higher accuracy of Google Translate for widely spoken languages and better preservation of original meanings.
Limitations and Future Work
The primary limitation identified is the degradation of performance in non-English languages, largely attributed to the accuracy of machine translation and the potential for altered meanings. Additionally, differences in topics and cultural contexts between languages pose a challenge.
Future work could involve:
- Enhancing the English baseline with even more diverse meme data.
- Investigating better machine translation models or techniques specifically for meme content.
- Developing language-specific models or incorporating domain-specific knowledge (e.g., political or diplomatic topics relevant to specific countries) to improve performance on non-English memes.
Summary updated