TL;DR
This paper presents a top-performing system for detecting propaganda techniques in text, specifically within internet memes. The authors leverage ensembling of multiple pre-trained transformer models and explore data augmentation techniques to improve performance on this challenging task.
What Problem Does This Paper Address?
Internet memes have become a powerful medium for spreading ideas and ideologies, often employing persuasive techniques that can be subtle and difficult to detect. SemEval-2021 Task 6 aimed to address this by focusing on the "detection of persuasion techniques in text and images." Specifically, this paper tackles subtask 1, which involves identifying propaganda techniques solely from the textual content of memes. The challenge lies in the nuanced nature of these techniques and the often informal, context-dependent language used in memes, making automated detection a significant hurdle.
Methods
The core of the LeCun system is an ensemble of five pre-trained transformer models: one DeBERTa large model and four RoBERTa large models. These models were trained using two primary approaches: a standard fine-tuning approach and a two-iteration approach. The two-iteration approach involved an initial training phase on a pre-processed dataset, followed by fine-tuning the resulting model on the competition dataset.
Data augmentation played a crucial role in enhancing the training data. Techniques like Synonym Replacement, Random Insertion, Random Swap, and Random Deletion were applied to an external dataset, while Back-translation was used for the competition dataset. This augmentation significantly increased the dataset size, which is particularly beneficial for smaller datasets.
Datasets
The system was trained and evaluated using two main datasets:
- Competition Dataset: This dataset, provided by SemEval-2021 Task 6, consists of short text samples extracted from memes.
- External Dataset: This dataset, sourced from a previous SemEval competition, comprises news articles containing propaganda spans. To make it compatible, the news articles were truncated to meme-like text fragments, and only the propaganda-containing parts were used.
The combination of these datasets, along with data augmentation, resulted in an "Augmented Dataset" of 52,966 samples.
Key Insights and Ideas
- Ensembling is Key: The paper strongly suggests that ensembling multiple transformer models, even those trained on similar data or with similar preprocessing, can lead to improved performance.
- Fine-tuning on Competition Data Boosts Performance: Fine-tuning models on the specific competition dataset after initial training on other datasets consistently improved F1-micro scores by up to 0.1.
- Data Augmentation's Impact: While not explicitly quantified in terms of direct comparison, the paper highlights the use of data augmentation as a strategy to increase dataset size, which is generally beneficial for model performance, especially with limited data.
- External Data Value: The inclusion of an external dataset, even after adaptation, contributed to the overall performance, suggesting the value of diverse data sources.
Key Results
The LeCun system achieved strong results on the SemEval-2021 Task 6:
- Test Dataset: An F1-micro score of 0.512.
- Development Dataset: An F1-micro score of 0.647.
The paper also reports individual model performances, with some RoBERTa models achieving F1-micro scores as high as 0.602 and 0.62 on the development set before ensembling. The final ensemble model (G) achieved the highest F1-micro score of 0.64.
Limitations and Future Work
The authors acknowledge several limitations and propose future research directions:
- "None" Class Performance: The model struggled with correctly classifying samples that did not contain any propaganda techniques (the "None" class). This is attributed to the sparsity of labels and the dominance of the "None" label in the dataset.
- Data Sparsity: The imbalance and sparsity of labels in the dataset are identified as a challenge.
- Future Work:
- Investigating the precise influence of data augmentation and external datasets on model performance.
- Exploring methods to improve the model's weaknesses, particularly in handling the "None" class.
- Examining how span identification (identifying the exact text segment containing propaganda) could further enhance technique classification.
- Conducting more in-depth experiments with different augmentation techniques and model architectures.
- Comparing models trained solely on the competition dataset versus those trained on the external dataset.
Summary updated