Abstract
Propaganda detection is a critical challenge in the modern information landscape, requiring the identification of subtle rhetorical strategies designed to manipulate public opinion. This paper presents UMSIForeseer, a system developed for SemEval-2020 Task 11 (Technique Classification). The system addresses the significant challenge of extreme class imbalance by combining fine-tuned BERT-base models with a balanced bagging ensemble strategy. Our approach demonstrates that while resampling improves the detection of rare "long-tail" propaganda techniques, balancing the trade-off between class-specific recall and overall precision remains a key hurdle.
Introduction: The Challenge of Automated Propaganda Detection
Propaganda is often designed to bypass critical thinking by leveraging emotional triggers or logical fallacies. Detecting it automatically requires more than just identifying "fake news"; it involves pinpointing specific rhetorical techniques used to sway the reader. SemEval-2020 Task 11 focused on classifying 14 such techniques, including "Loaded Language," "Slogans," and "Appeal to Fear."
The primary technical difficulty in this task is the extreme imbalance of the dataset. Common techniques like "Loaded Language" appear thousands of times, whereas rare techniques like "Bandwagon" or "Thought-Terminating Cliches" appear in fewer than 80 instances. Standard machine learning models often optimize for overall accuracy by ignoring these minority classes, leading to a "blind spot" for less frequent but equally impactful forms of propaganda.
Methodology: UMSIForeseer Architecture
The UMSIForeseer system relies on two core components: fine-tuned BERT representations and an ensemble learning strategy designed for imbalanced data.
BERT-base Fine-tuning
The team utilized the BERT-base-uncased model as the primary feature extractor and classifier. Textual fragments were preprocessed into sequences of 64 tokens, including the standard [CLS] and [SEP] tokens. The model was fine-tuned for 4 epochs, allowing it to adapt its pre-trained linguistic knowledge to the specific nuances of propagandistic language.
Balanced Bagging and Resampling
To mitigate the class imbalance problem, the researchers implemented a "balanced bagging" strategy. This involved:
- Resampling: A target of 400 examples per class was established. Minority classes were oversampled using bootstrapping, while majority classes were undersampled to reach this threshold.
- Bagging (Bootstrap Aggregating): Nine separate BERT models were trained, each on a different balanced subset of the data.
- Ensemble Voting: The final classification for any given text fragment was determined by a majority vote across the nine models.
Datasets and Experimental Setup
The system was trained and evaluated on the SemEval-2020 Task 11 dataset, which includes 8,982 news articles. The training set consisted of 6,129 labeled examples across 14 techniques. Evaluation was performed on a validation set (1,063 examples) and a final test set (1,790 examples).
Results and Discussion
The UMSIForeseer system achieved a competitive ranking, placing 17th out of 46 participating teams. The top-performing configuration—a single BERT model trained on the original unbalanced data—achieved a Testing F1-score of 0.5514.
The Impact of Bagging
The bagging approach provided a significant breakthrough for rare classes. On the original unbalanced data, the model's F1-score for classes like "Black and White Fallacy" or "Appeal to Authority" was 0.0000, meaning the model never predicted them. The balanced ensemble successfully raised the F1-score for "Appeal to Authority" to 0.1290. However, this improvement in the "long-tail" came at a slight cost to overall performance, as the ensemble was slightly less precise on the most frequent classes compared to the single unbalanced model.
Conclusion
The UMSIForeseer system demonstrates that BERT is a powerful tool for propaganda detection, even with minimal hyperparameter tuning. While balanced bagging is effective at "unmasking" rare propaganda techniques that standard models ignore, future work should focus on more sophisticated resampling methods like SMOTE and advanced ensemble techniques like boosting to further refine the balance between rare-class detection and overall accuracy.
Summary updated