Transformers at SemEval-2020 Task 11: Propaganda Fragment Detection Using Diversified BERT Architectures Based Ensemble Learning

Ekansh Verma, Vinodh Motupalli, Souradip ChakrabortySemEval @ COLING 2020 · OpenReview

TL;DR

This paper tackles the challenging task of automatically identifying propaganda within news articles by treating it as a word-level binary classification problem. The authors propose a sophisticated ensemble of fine-tuned BERT models, incorporating techniques like multi-sample dropout and convolutional layers, to achieve state-of-the-art performance and were recognized among the top 10 teams in the SemEval-2020 competition.

What Problem Does This Paper Address?

Identifying propaganda in news articles is crucial for fostering media literacy and promoting more rational discourse. Propaganda, defined as information spread with the intention of influencing opinions, can be subtle and embedded within text. Automatically detecting these "propaganda spans" is a complex NLP task because it requires understanding not just individual words but also their contextual usage and the persuasive intent behind them. This paper specifically addresses the "span identification" sub-task of SemEval-2020 Task 11, which aims to pinpoint the exact text segments containing propaganda techniques.

Methods

The core of the authors' approach is an ensemble learning framework built upon diversified BERT architectures. They treat propaganda span identification as a word-level binary classification problem.

  • BERT Fine-tuning: They leverage pre-trained BERT models (BERT base, BERT large, and RoBERTa base) and fine-tune them for this specific task.
  • Diversified Architectures: To enhance detection capabilities, they experiment with several modifications:
  • BERT-Linear: A standard BERT model with a linear classification layer and softmax activation on top.
  • BERT-CNN: Integrates a 1D convolutional neural network (CNN) on top of BERT to capture local patterns in the contextualized embeddings.
  • Multi-Sample Dropout: Implements an enhanced dropout technique with multiple dropout masks applied to different subsets of neurons, aiming to improve generalization and stabilize training.
  • Ensemble Learning: The final model is an ensemble of these diversified BERT-based models. The predictions from individual models are combined to produce a more robust final prediction.
  • Loss Functions: Given the imbalanced nature of propaganda data (many more non-propaganda instances), they employ a weighted cross-entropy loss function, assigning higher weights to propaganda instances to ensure they are adequately learned. They also experimented with focal loss but found it didn't yield significant improvements in their setup.

Datasets

The research utilizes the dataset provided for SemEval-2020 Task 11. This dataset consists of news articles in plain text format, with propaganda spans annotated at the character level. The authors converted these character-level annotations to word-level tokens for their models.

  • Training Set: 371 articles
  • Development Set: 75 articles
  • Test Set: 90 articles

The dataset contains approximately 5468 propaganda spans across 344,095 total words.

Key Insights and Ideas

  • Ensembling is Key: The most significant insight is that combining multiple, diversified BERT-based models (an ensemble) yields superior performance compared to any single model. This highlights the benefit of capturing different aspects of language and propaganda through varied architectural choices.
  • CNNs Aid Local Pattern Detection: Integrating CNNs on top of BERT proved beneficial for identifying local patterns within the text, suggesting that these local features are important for pinpointing propaganda spans.
  • Multi-Sample Dropout for Robustness: The multi-sample dropout technique helped in stabilizing training and improving the generalizability of the models, a common challenge in deep learning.
  • Data Imbalance Requires Careful Handling: The inherent imbalance in propaganda detection datasets necessitates specialized loss functions (like weighted cross-entropy) to prevent models from simply predicting the majority class.

Key Results

The authors achieved an F1-score of 0.481 on the test dataset with their BERT-Ensemble model. This performance placed them among the top 10 teams in the SemEval-2020 propaganda span identification challenge.

  • The best single model, BERT-Linear, achieved an F1 score of 0.428 on the development set and 0.469 on the test set.
  • The BERT-Ensemble model demonstrated the highest performance, with an F1 score of 0.437 on the development set and 0.481 on the test set.

Limitations and Future Work

  • Limitations: The authors acknowledge that their experiments with focal loss did not yield improvements, suggesting that suitable hyperparameter tuning might be necessary for it to be effective. They also note that their current approach relies on labeled data.
  • Future Work: The authors plan to explore semi-supervised learning paradigms to train models with less labeled data. They also express interest in investigating FLAIR embeddings and the T5 architecture, which have shown promise in related entity extraction tasks.

Summary updated

BibTeX
@inproceedings{verma2020transformers,
  author = {Ekansh Verma and Vinodh Motupalli and Souradip Chakraborty},
  title = {{Transformers at SemEval-2020 Task 11: Propaganda Fragment Detection Using Diversified BERT Architectures Based Ensemble Learning}},
  booktitle = {SemEval @ COLING},
  year = {2020},
  url = {https://openreview.net/forum?id=ESMRFU4aH7O},
}
Figure