TL;DR
This paper explores using Multi-Task Learning (MTL) to improve propaganda detection in memes, a challenging task due to limited labeled data. By training a BERT-based model on related tasks like single-label propaganda detection and fake news credibility assessment, they found that while MTL can accelerate learning, it doesn't consistently outperform a strong single-task BERT model on the target task.
What Problem Does This Paper Address?
Online misinformation is a growing concern, and propaganda detection is a crucial part of combating it. Specifically, identifying propaganda techniques at a fine-grained, word-level is difficult because it requires extensive manual annotation, which is labor-intensive and costly to produce. This scarcity of labeled data makes it hard to train effective machine learning models. The authors aim to address this data sparsity by leveraging related, larger datasets through Multi-Task Learning (MTL).
Methods
The core of the proposed solution is a BERT-based architecture. The authors experiment with different MTL strategies:
- Single Task: A standard BERT-Base-Uncased model with a token-level multi-label classification layer, trained solely on the SemEval-2021 Task 6 (S21) propaganda dataset.
- Sequential Multi-Task Learning (MULTI-S): The BERT model is first pre-trained on an auxiliary task (either SemEval-2020 Task 11 propaganda or a fake news credibility dataset) and then fine-tuned on the S21 dataset.
- Parallel Multi-Task Learning (MULTI-P): The BERT model is trained jointly on the S21 dataset and an auxiliary dataset simultaneously.
The architecture includes classification modules that take BERT's token-level or document-level representations and produce scores for different classification objectives.
Datasets
The paper utilizes three datasets:
- SemEval-2021 Task 6 (S21): The primary dataset for the target task, containing 870 memes annotated with 20 propaganda techniques at the word level. This dataset is relatively small, especially its development (63 documents) and test sets.
- SemEval-2020 Task 11 (S20): A dataset of 446 press articles annotated with 14 propaganda categories at the word level. This dataset is used as an auxiliary task.
- Fake News Credibility (FN): A dataset of 103,219 news articles, from which 50,000 sentences with binary credibility labels (credible/non-credible) were sampled. This dataset is also used as an auxiliary task.
Key Insights and Ideas
- Leveraging Related Tasks: The fundamental idea is that knowledge gained from related tasks (like general propaganda detection or fake news classification) can help a model learn the target task (fine-grained propaganda detection in memes) more effectively, especially when labeled data for the target task is scarce.
- MTL's Impact on Learning Speed: Pre-training with auxiliary tasks (sequential MTL) significantly accelerates the learning process. Models trained with auxiliary data reach a good performance level much faster than the single-task model.
- Data Domain Mismatch: The authors found that the linguistic constructions in press articles (S20) and memes (S21) are quite different. This domain mismatch limits the benefits of using S20 as an auxiliary task for specific propaganda techniques.
- No Clear Superiority of MTL: Despite the potential benefits, the MTL approaches did not consistently outperform the single-task BERT model, especially on the test set. In some cases, the single-task model even performed better after extended training.
Key Results
- Development Set Performance: The MULTI-P S20-S21 model achieved the highest F1 score on the development set. This model was used for the official submission to the shared task.
- Test Set Performance: Surprisingly, the single-task S21 model provided the best performance on the test set.
- Learning Acceleration: Sequential MTL with S20 data allowed the model to reach an F1 score of 0.4 approximately 7 epochs earlier compared to the single-task approach, which took around 14 epochs.
- Per-Technique Performance: When using S20 as an auxiliary task, performance improved for some techniques (e.g., Smears), but dropped for others that were present in both datasets (e.g., Appeal to authority, Slogans). The relationship with fake news detection (FN) was even weaker, with many techniques not being recognized.
Limitations and Future Work
- Limited Link Strength: The authors hypothesize that the link between the main propaganda detection task and the auxiliary tasks might not be strong enough to yield significant benefits from MTL.
- Domain Differences: The significant differences between the domains of the auxiliary datasets (press articles, general news) and the target dataset (memes) likely hinder the transfer of knowledge.
- Future Directions: The authors suggest extending the auxiliary task portfolio with corpora that reflect other related issues, such as hate speech or hyper-partisan language, to explore MTL's relevance further. They also acknowledge that propaganda is a complex phenomenon, and different techniques might require tailored approaches.
Summary updated