Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic

Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, Preslav Nakov

Abstract

Propaganda is the expression of an opinion or an action by an individual or a group deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends, which is achieved by means of well-defined rhetorical and psychological devices. Propaganda techniques are commonly used in social media to manipulate or to mislead users. Thus, there has been a lot of recent research on automatic detection of propaganda techniques in text as well as in memes. However, so far the focus has been primarily on English. With the aim to bridge this language gap, we ran a shared task on detecting propaganda techniques in Arabic tweets as part of the WANLP 2022 workshop, which included two subtasks. Subtask 1 asks to identify the set of propaganda techniques used in a tweet, which is a multilabel classification problem, while Subtask 2 asks to detect the propaganda techniques used in a tweet together with the exact span(s) of text in which each propaganda technique appears. The task attracted 63 team registrations, and eventually 14 and 3 teams made submissions for subtask 1 and 2, respectively. Finally, 11 teams submitted system description papers.

Introduction

While automatic propaganda detection has seen significant progress recently, the vast majority of research and available datasets have focused on English. Propaganda is not just about "falsehoods"; it is about the specific rhetorical and psychological techniques used to manipulate an audience, ranging from emotional appeals (e.g., fear, flag-waving) to logical fallacies (e.g., whataboutism, red herrings). Arabic is one of the most widely spoken languages globally and has a massive presence on social media, yet it lacks the specialized benchmarks needed to train models to recognize these subtle manipulative tactics. The WANLP 2022 Shared Task was established to bridge this gap by providing a curated dataset and a platform for researchers to test Arabic-specific NLP models.

The Shared Task and Dataset

The shared task was split into two distinct challenges:

  1. Subtask 1 (Multilabel Classification): Given a tweet, identify which of the 20 propaganda techniques are present.
  2. Subtask 2 (Span Detection): Identify the specific techniques and the exact character spans (fragments of text) where they occur.
The taxonomy of 20 propaganda techniques used for labeling the Arabic tweet dataset.

The authors introduced a new dataset consisting of 930 Arabic tweets sampled from top news sources in every Arab country and major international Arabic news outlets. Three independent annotators labeled the text at the fragment level, followed by a consolidation phase to ensure high-quality "gold" labels. The taxonomy included 20 techniques, though the final annotated set was highly skewed; "Loaded Language" and "Name Calling/Labeling" were very frequent, while others like "Bandwagon" or "Straw Man" had zero occurrences in this specific sample, highlighting the "long tail" challenge of real-world propaganda.

Methodology

Participating teams primarily utilized Transformer-based architectures. Models specifically pre-trained on Arabic text, such as AraBERT, MARBERT, and AraELECTRA, were the most popular and effective. To handle the difficulty of span detection in Subtask 2, top-performing teams combined these transformers with Conditional Random Fields (CRF) or treated the problem as a Named Entity Recognition (NER) task.

The annotation interface showing a tweet in Arabic, its translation, and the specific spans identified for propaganda techniques.

Some teams also experimented with data augmentation, such as translating existing English propaganda datasets into Arabic to increase their training pool. However, results were mixed, suggesting that propaganda techniques may be expressed differently across cultures and languages, making simple translation insufficient.

Key Results and Insights

The task demonstrated a significant performance gap between identifying that a tweet contains propaganda and identifying exactly which words are propagandistic. Subtask 1 saw F1 scores around 0.65, while Subtask 2 peaked at 0.39.

  • Subtask 1 Winner: Team NGU_CNLP achieved the highest Micro-F1 of 0.649 using an ensemble of AraBERT models and translation-based data augmentation.
  • Subtask 2 Winner: Team Pythoneers led with a Micro-F1 of 0.396, utilizing AraBERT combined with a CRF layer to better handle sequence labeling.

Key insights from the competition include the superiority of domain-specific pre-training (AraBERT significantly outperformed general models like mBERT) and the primary hurdle of data imbalance. Models struggle to learn rare techniques when the dataset is dominated by common ones like "Loaded Language."

Conclusion and Future Work

The WANLP 2022 Shared Task represents a critical step in expanding propaganda detection to the Arabic language. Future work will focus on scaling the dataset to better represent the "long tail" of techniques, moving toward hierarchical modeling to help models generalize better, and expanding the scope to include multimodal detection (images and memes), as social media propaganda is rarely text-only.

Summary updated

Abstract

Propaganda is the expression of an opinion or an action by an individual or a group deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends, which is achieved by means of well-defined rhetorical and psychological devices. Propaganda techniques are commonly used in social media to manipulate or to mislead users. Thus, there has been a lot of recent research on automatic detection of propaganda techniques in text as well as in memes. However, so far the focus has been primarily on English. With the aim to bridge this language gap, we ran a shared task on detecting propaganda techniques in Arabic tweets as part of the WANLP 2022 workshop, which included two subtasks. Subtask~1 asks to identify the set of propaganda techniques used in a tweet, which is a multilabel classification problem, while Subtask~2 asks to detect the propaganda techniques used in a tweet together with the exact span(s) of text in which each propaganda technique appears. The task attracted 63 team registrations, and eventually 14 and 3 teams made submissions for subtask 1 and 2, respectively. Finally, 11 teams submitted system description papers.
BibTeX
@inproceedings{alam2022overview,
  author = {Firoj Alam and Hamdy Mubarak and Wajdi Zaghouani and Giovanni Da San Martino and Preslav Nakov},
  title = {{Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic}},
  booktitle = {WANLP @ EMNLP},
  year = {2022},
  url = {https://openreview.net/forum?id=vZyECuwGolw},
}
Figure