Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic

Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, Preslav Nakov

Abstract

This paper summarizes the findings of the WANLP 2022 Shared Task, the first competition dedicated to detecting fine-grained propaganda techniques in Arabic social media. The task comprised two challenges: multi-label classification of tweets and the detection of specific text spans associated with 20 distinct rhetorical techniques. Using a newly curated dataset of 930 tweets, participants demonstrated that while Arabic-specific Transformer models like AraBERT excel at classification, identifying the exact boundaries of propagandistic language remains a complex problem.

Introduction

Computational propaganda—the use of rhetorical and psychological devices to manipulate public opinion—has become a pervasive issue on social media. While significant research has focused on English-language propaganda, Arabic remains under-addressed despite its global importance and the complexity of its linguistic structure. The WANLP 2022 Shared Task was established to bridge this gap, providing a benchmark for identifying 20 specific propaganda techniques, such as "loaded language," "fear-mongering," and "whataboutism," in Arabic tweets.

Methodology and Task Design

The shared task was organized into two primary subtasks to evaluate different levels of detection granularity:

  1. Subtask 1 (Multi-label Classification): Participants were required to identify which of the 20 predefined propaganda techniques were present in a given tweet.
  2. Subtask 2 (Span Detection): This more granular task required identifying both the technique and the specific character-level spans within the text where the technique was applied.
Example of tweet annotation for propaganda techniques, illustrating both label assignment and span identification.

Most participating teams employed Transformer-based architectures. Models pre-trained specifically on Arabic corpora, such as AraBERT, MARBERT, and ARBERT, were the most effective. Advanced strategies included data augmentation via translation of English datasets, ensemble modeling, and the use of Conditional Random Fields (CRF) for sequence labeling in Subtask 2.

Dataset Description

The researchers developed a novel dataset consisting of 930 tweets sampled from 20 major Arabic news outlets (e.g., Al Arabiya, Al Jazeera). The annotation process involved three independent annotators per tweet, followed by a consolidation phase to establish a "gold standard."

The taxonomy of 20 techniques covers a broad spectrum of rhetorical devices. However, the distribution of these techniques in the dataset was highly skewed. "Loaded Language" and "Name Calling/Labeling" were the most frequent, while more complex logical fallacies like "Straw Man" or "Reductio ad Hitlerum" were absent, reflecting the nature of short-form social media communication.

Results and Discussion

The competition saw participation from 14 teams in Subtask 1 and 3 teams in Subtask 2.

  • Subtask 1: The top-performing system (NGU_CNLP) achieved a Micro-F1 score of 0.649 using an ensemble of AraBERT models and translation-based data augmentation.
  • Subtask 2: The winning team (Pythoneers) reached a Micro-F1 of 0.396, highlighting the significant difficulty of precise span detection compared to tweet-level classification.

A key finding was the superiority of Arabic-specific pre-training. Models like AraBERT consistently outperformed multilingual counterparts like mBERT or XLM-R, suggesting that capturing the nuances of Arabic morphology and dialect is crucial for detecting subtle rhetorical patterns.

Conclusion

The WANLP 2022 Shared Task established a vital baseline for Arabic propaganda detection. While classification performance is promising, the lower scores in span detection indicate that the boundaries of rhetorical devices are often context-dependent and fuzzy. Future work should focus on expanding the dataset to include rare techniques, exploring hierarchical classification, and incorporating multimodal data such as memes where text and imagery interact.

Summary updated

Abstract

Propaganda is the expression of an opinion or an action by an individual or a group deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends, which is achieved by means of well-defined rhetorical and psychological devices. Propaganda techniques are commonly used in social media to manipulate or to mislead users. Thus, there has been a lot of recent research on automatic detection of propaganda techniques in text as well as in memes. However, so far the focus has been primarily on English. With the aim to bridge this language gap, we ran a shared task on detecting propaganda techniques in Arabic tweets as part of the WANLP 2022 workshop, which included two subtasks. Subtask~1 asks to identify the set of propaganda techniques used in a tweet, which is a multilabel classification problem, while Subtask~2 asks to detect the propaganda techniques used in a tweet together with the exact span(s) of text in which each propaganda technique appears. The task attracted 63 team registrations, and eventually 14 and 3 teams made submissions for subtask 1 and 2, respectively. Finally, 11 teams submitted system description papers.
BibTeX
@inproceedings{alam2022overview,
  author = {Firoj Alam and Hamdy Mubarak and Wajdi Zaghouani and Giovanni Da San Martino and Preslav Nakov},
  title = {{Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic}},
  booktitle = {WANLP @ EMNLP},
  year = {2022},
  url = {https://openreview.net/forum?id=vZyECuwGolw},
}
Figure