TTUI at SemEval-2020 Task 11: Propaganda Detection with Transfer Learning and Ensembles

Moonsung Kim, Steven BethardSemEval @ COLING 2020 · DOI

Abstract

In this paper, we describe our approaches and systems for the SemEval-2020 Task 11 on propaganda technique detection. We fine-tuned BERT and RoBERTa pre-trained models then merged them with an average ensemble. We conducted several experiments for input representations dealing with long texts and preserving context as well as for the imbalanced class problem. Our system ranked 20th out of 36 teams with 0.398 F1 in the SI task and 14th out of 31 teams with 0.556 F1 in the TC task.

Introduction

Propaganda is a subtle art that employs psychological and rhetorical techniques to influence audiences, often masking its intentions within standard news reporting. Computational detection is challenging due to context sensitivity, the need for fine-grained span identification (SI) and technique classification (TC), and significant data imbalance. This paper presents the TTUI system, which leverages transfer learning and ensembles to address these challenges.

Methodology

The system utilizes an ensemble of fine-tuned BERT and RoBERTa models. Key technical innovations include specialized input representations and post-processing heuristics.

Overlapped Sentence Chunking

To handle the 512-token limit of transformer models while maintaining context, the authors implemented an overlapping sentence chunking strategy. By splitting articles into chunks that overlap by four sentences, the model retains sufficient surrounding information to determine if specific text fragments constitute propaganda.

Comparison of sentence chunking strategies, highlighting the overlapped approach used to preserve context.

Ensembling and Post-processing

The final system uses an average ensemble of BERT-large and RoBERTa-large models. To improve span continuity, a "gap-filling" heuristic was applied: if words immediately before and after a target word were identified as propaganda, the target word was also labeled as such. This simple post-processing step improved the SI F1 score from 0.449 to 0.454 on the development set.

Results and Analysis

The system was evaluated on the Propaganda Techniques Corpus (PTC), consisting of 446 news articles annotated for 14 techniques.

Performance Comparison

In the SI task, the TTUI system achieved an F1 score of 0.398. Notably, the model demonstrated significantly higher precision (0.669) than the top-ranked system (0.565), although it suffered from lower recall, indicating a conservative prediction strategy.

Performance comparison of the TTUI system against the top-ranked system on development and test sets.

The Context Paradox

A significant finding was that while context is vital for span identification, it can be detrimental to classification. The best results for the TC task (0.534 F1) were obtained using only the propaganda fragment. The authors suggest that surrounding text often contains other propaganda techniques, which act as noise and degrade classifier performance.

Conclusion

The TTUI system highlights the strengths of transformer ensembles in propaganda detection. While the approach achieved high precision, future work will focus on improving recall and addressing performance on underrepresented propaganda techniques through advanced data augmentation or few-shot learning.

Summary updated

BibTeX
@inproceedings{kim2020ttui,
  author = {Moonsung Kim and Steven Bethard},
  title = {{TTUI at SemEval-2020 Task 11: Propaganda Detection with Transfer Learning and Ensembles}},
  booktitle = {SemEval @ COLING},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.240},
}
Figure