Abstract
In this paper, we describe our approaches and systems for the SemEval-2020 Task 11 on propaganda technique detection. We fine-tuned BERT and RoBERTa pre-trained models then merged them with an average ensemble. We conducted several experiments for input representations dealing with long texts and preserving context as well as for the imbalanced class problem. Our system ranked 20th out of 36 teams with 0.398 F1 in the SI task and 14th out of 31 teams with 0.556 F1 in the TC task.
Introduction
Propaganda is a subtle art that employs psychological and rhetorical techniques to influence audiences, often masking its intentions within standard news reporting. Computational detection is challenging due to context sensitivity, the need for fine-grained span identification (SI) and technique classification (TC), and significant data imbalance. This paper presents the TTUI system, which leverages transfer learning and ensembles to address these challenges.
Methodology
The system utilizes an ensemble of fine-tuned BERT and RoBERTa models. Key technical innovations include specialized input representations and post-processing heuristics.
Overlapped Sentence Chunking
To handle the 512-token limit of transformer models while maintaining context, the authors implemented an overlapping sentence chunking strategy. By splitting articles into chunks that overlap by four sentences, the model retains sufficient surrounding information to determine if specific text fragments constitute propaganda.
Ensembling and Post-processing
The final system uses an average ensemble of BERT-large and RoBERTa-large models. To improve span continuity, a "gap-filling" heuristic was applied: if words immediately before and after a target word were identified as propaganda, the target word was also labeled as such. This simple post-processing step improved the SI F1 score from 0.449 to 0.454 on the development set.
Results and Analysis
The system was evaluated on the Propaganda Techniques Corpus (PTC), consisting of 446 news articles annotated for 14 techniques.
Performance Comparison
In the SI task, the TTUI system achieved an F1 score of 0.398. Notably, the model demonstrated significantly higher precision (0.669) than the top-ranked system (0.565), although it suffered from lower recall, indicating a conservative prediction strategy.
The Context Paradox
A significant finding was that while context is vital for span identification, it can be detrimental to classification. The best results for the TC task (0.534 F1) were obtained using only the propaganda fragment. The authors suggest that surrounding text often contains other propaganda techniques, which act as noise and degrade classifier performance.
Conclusion
The TTUI system highlights the strengths of transformer ensembles in propaganda detection. While the approach achieved high precision, future work will focus on improving recall and addressing performance on underrepresented propaganda techniques through advanced data augmentation or few-shot learning.
Summary updated