Team DiSaster at SemEval-2020 Task 11: Combining BERT and Hand-crafted Features for Identifying Propaganda Techniques in News

Anders Kaas, Viktor Torp Thomsen, Barbara Plank

Introduction

Propaganda detection has traditionally focused on document-level classification. However, SemEval-2020 Task 11 introduced the challenge of fine-grained, span-level identification. This task requires models to not only detect propagandistic content but also identify the specific technique used—such as "Loaded Language," "Slogans," or "Repetition"—within localized text fragments. Team DiSaster addressed this by developing a hybrid model that combines the deep contextual representations of BERT with engineered linguistic features designed to capture global, article-level context.

Methodology

The authors implemented a pipeline called DiSaster based on stacked generalization. This architecture uses a two-tier approach to merge disparate signals from deep learning and traditional feature engineering.

The Base Learners

  1. BERT-Large: Fine-tuned on specific propaganda spans to generate a 14-dimensional vector of prediction scores (logits).
  2. Hand-crafted Features: The team engineered 54 features to capture information often missed by localized Transformer windows. Key features included aowc (Article One Word Counter), which tracks word stem frequency across the entire article, and assc (Article Span Sentence Counter), which identifies repeated multi-word spans.
  3. Logistic Regression: A traditional model trained exclusively on these hand-crafted features to provide a baseline statistical signal.

The Meta-Classifier

The outputs from the base learners—the BERT logits, the raw hand-crafted features, and the Logistic Regression scores—are concatenated and fed into a 3-layer Feed-Forward Neural Network (FFNN). This meta-classifier makes the final determination across the 14 consolidated propaganda classes.

Figure 1: The DiSaster model architecture utilizing stacked generalization to combine BERT and hand-crafted features.

Dataset and Task Setup

The study utilized the SemEval-2020 Task 11 dataset, comprising 371 news articles. While the original data contained 18 techniques, they were consolidated into 14 classes for the competition. The dataset exhibited a significant "long-tail" distribution; the top four classes accounted for over 70% of the training data, presenting a substantial challenge for minority class classification.

Key Results and Ablation Study

The DiSaster system achieved a micro-average F1 score of 0.628 on the development set and 0.566 on the official test set, placing 11th out of 32 participating teams.

An extensive ablation study confirmed that while BERT is the primary driver of performance, hand-crafted features provide critical boosts for specific techniques. For instance, removing BERT caused the F1 score to drop precipitously from 0.672 to 0.443. Conversely, the inclusion of the aowc feature was vital for the "Repetition" class; without it, the model struggled to recognize spans repeated outside the immediate BERT context window.

Table 1: Performance metrics and ablation study results across various propaganda techniques and model configurations.

Discussion and Insights

The research highlights a specific limitation in standard Transformer models: "nearsightedness." Because BERT typically processes text in isolated spans, it lacks the global context necessary to identify techniques like "Repetition" or "Flag-Waving" that rely on document-wide patterns. The authors demonstrated that feature engineering remains highly relevant in the era of Large Language Models (LLMs) when focused on these global dependencies.

Furthermore, the team observed that propaganda techniques are not independent; the presence of one technique often correlates with the presence of others within the same article. Future work could involve using Recurrent Neural Networks (RNNs) or more sophisticated attention mechanisms to better model these inter-span relationships across entire documents.

Summary updated

BibTeX
@inproceedings{kaas2020team,
  author = {Anders Kaas and Viktor Torp Thomsen and Barbara Plank},
  title = {{Team DiSaster at SemEval-2020 Task 11: Combining BERT and Hand-crafted Features for Identifying Propaganda Techniques in News}},
  booktitle = {SemEval @ COLING},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.238},
}
Figure