TL;DR
This paper tackles the challenging task of automatically detecting persuasion techniques in multilingual memes, a crucial aspect of combating online disinformation. The authors propose a multimodal approach combining advanced text and image preprocessing with pre-trained language and vision models, achieving competitive results by effectively handling class imbalance and multilingualism.
What Problem Does This Paper Address?
The proliferation of memes on social media presents a unique challenge for disinformation detection. Memes, with their blend of text and imagery, can subtly embed persuasive techniques and rhetorical strategies to influence public opinion. Identifying these techniques across multiple languages is essential for understanding and combating online propaganda campaigns. SemEval-2024 Task 4 specifically focuses on this problem, aiming to develop models that can accurately detect persuasion techniques in memes in languages like English, Bulgarian, and North Macedonian. The core challenge lies in effectively processing and integrating information from both textual and visual modalities, while also dealing with noisy, often informal, meme text and potential class imbalances.
Methods
The authors employed a multimodal approach, leveraging pre-trained models for both text and image feature extraction.
- Text Preprocessing: A significant effort was dedicated to cleaning and standardizing the textual content extracted from memes. This involved using the OpenAI API for initial cleaning (removing extraneous information like dates and usernames) and the NLTK library for further normalization (tokenization, stemming, stop-word removal). Crucially, they also incorporated a manual data correction step to address discrepancies between the extracted text and the actual text in the meme image, which proved vital for improving accuracy.
- Image Preprocessing: Standard techniques like resizing and normalization were applied to image data using the PyTorch framework.
- Feature Extraction: For text, they fine-tuned pre-trained language models like XLM-ROBERTa and GPT-2. For images, they explored Convolutional Neural Networks (CNNs) such as VGG and Vision Transformers (ViTs).
- Multimodal Fusion: The extracted text and image features were concatenated to create a unified multimodal representation.
- Model Architecture: A classification layer was built on top of this fused representation. Their best-performing model for Subtask 2b combined VGG-16 for image features and GPT-2 for text features.
- Training: To address class imbalance, they used Focal Loss in conjunction with AdamW optimizer. Hyperparameters like learning rate, batch size, and dropout rate were tuned using grid search and cross-validation.
Datasets
The primary dataset used was the SemEval-2024 Task 4 dataset, specifically for Subtask 2b.
- Training Data: 1200 samples.
- Validation Data: 150 samples.
- Development Data: Initially unlabeled, later provided as "dev_gold_labels" with 600 samples.
- Test Data: Not explicitly detailed in terms of size, but results were reported on a test set across three languages.
The paper notes that initially, the development dataset labels were unavailable, and they later used the "gold_unlabeled" dataset for validation. They also mention that the training data distribution across classes was not uniform, necessitating specific loss functions.
Key Insights and Ideas
- Manual Text Correction is Crucial: The authors discovered that automated text extraction from memes is often noisy and inaccurate. Manual correction of textual discrepancies between extracted text and the image content significantly improved their model's performance.
- Multilingual Models are Key: Instead of translating all meme text to English, they found that using pre-trained multilingual models like GPT and XLM-ROBERTa was more effective for handling the diverse languages in the dataset.
- Focal Loss for Class Imbalance: The non-uniform distribution of "propagandistic" vs. "non-propagandistic" memes made class imbalance a significant issue. Focal Loss proved effective in mitigating this.
- GPT-2 Shows Strong Generalizability: The paper suggests that GPT-2, compared to XLM-ROBERTa, exhibits better generalizability, potentially due to its lower token limit.
Key Results
The authors' system achieved competitive results on the SemEval-2024 Task 4, Subtask 2b.
- On the English test dataset, their best model achieved a macro-F1 score of 0.67 and a micro-F1 score of 0.74.
- The best model combination for Subtask 2b was VGG-16 (image) + GPT-2 (text).
- Results on the test set across three languages showed:
- English: F1 macro 0.67398, F1 micro 0.74000
- Bulgarian: F1 macro 0.51637, F1 micro 0.74000
- North Macedonian: F1 macro 0.57653, F1 micro 0.79000
- These results significantly outperformed baseline models, especially in F1 micro scores.
Limitations and Future Work
- Limitations: The paper acknowledges that the initial model exhibited signs of overfitting without proper normalization, which they addressed with regularization techniques. The manual data correction process, while effective, is labor-intensive.
- Future Work: The authors suggest future research could focus on:
- Pre-training models specifically on meme-centric data.
- Refining preprocessing techniques for meme text.
- Further exploring the potential of large amounts of available data to improve model accuracy and robustness.
Summary updated