ZenPropaganda: A Comprehensive Study on Identifying Propaganda Techniques in Russian Coronavirus-Related Media

Anton Chernyavskiy, Svetlana Shomova, Irina Dushakova, Ilya Kiriya, Dmitry IlvovskyCOLING 2024

TL;DR

This paper introduces "ZenPropaganda," a novel, meticulously annotated dataset of Russian-language COVID-19-related texts from VKontakte and Yandex.Zen, designed to identify propaganda techniques. The authors also propose a new multi-level classification scheme and evaluate state-of-the-art models, highlighting the challenges and nuances of detecting propaganda in real-world social media content.

What Problem Does This Paper Address?

The proliferation of "fake news" and manipulative content online, especially amplified by social media and AI, poses a significant challenge to media literacy and informed public discourse. While propaganda detection is an active research area, there's a persistent need for more comprehensive datasets and refined classification schemes, particularly for non-English languages and specific domains like social media. Existing approaches often struggle with the fine-grained nature of propaganda techniques and the operationalization of these concepts for automated analysis. This paper addresses the gap by creating a specialized Russian dataset and a structured classification system to better understand and detect propaganda in a popular online context.

Methods

The core methodological contribution is the development of a novel multi-level classification scheme for propaganda techniques. This scheme categorizes techniques into three types:

  • Narrative techniques: Affecting the entire text, often harder to systematize and automate.
  • Fragment-level techniques: Targeting specific parts of the text.
  • Markers of manipulation: Surface-level indicators (e.g., excessive capitalization, exclamation marks) that can be automatically highlighted.

The authors also adapted and evaluated Transformer-based models, specifically RoBERTa, for two subtasks:

  • Span Identification (SI): Identifying the exact character spans within a text that contain propaganda. This involved using a BIO encoding scheme and incorporating a Conditional Random Field (CRF) layer for improved sequence labeling.
  • Technique Classification (TC): Assigning one or more propaganda technique labels to identified spans.

Datasets

The primary contribution is the ZenPropaganda dataset.

  • Source: Texts collected from VKontakte (user-generated content) and Yandex.Zen (blogging platform).
  • Content: COVID-19-related texts.
  • Annotation: Manually annotated with fine-grained labeling of manipulative spans.
  • Size: 125 texts, containing approximately 2400 manipulative segments.
  • Key Features: Texts are at least 1000 characters long, and annotations are character-level precise.

The authors also compared their dataset and findings with existing benchmarks, including:

  • SemEval-2020 Task 11: A dataset for propaganda detection in news articles. ZenPropaganda differs in its focus on social media and longer manipulative spans.
  • SemEval-2023 (multilingual): Another relevant dataset, though the authors note differences in subtasks and domains.

Key Insights and Ideas

  • Social Media Nuances: Propaganda on platforms like VKontakte and Yandex.Zen exhibits unique characteristics, including longer manipulative spans and a tendency for authors to repeatedly use a few core techniques.
  • Operationalization Challenges: Defining and consistently annotating propaganda techniques is difficult due to interpretive and terminological ambiguities, especially in Russian research.
  • Intra-text Focus: The research deliberately focuses on manipulation within texts, excluding external fact-checking, which simplifies the task but limits the scope.
  • Conspiracy Theories: A notable observation was the frequent use of conspiracy theories, suggesting this could be a valuable addition to future classification schemes.
  • Model Performance Gap: State-of-the-art models, while adapted, show significantly lower performance on this Russian social media dataset compared to English news datasets, highlighting the need for domain-specific and language-specific research.

Key Results

  • Dataset Creation: The ZenPropaganda dataset, with its fine-grained, character-level annotations, provides a valuable resource for Russian propaganda detection research.
  • Span Identification: The RoBERTa-CRF model achieved an F1-score of 0.30 for span identification on the test sample, which is lower than English benchmarks but considered reasonable given the dataset's specifics (longer spans, different classes). Sentence-level F1 was 0.43.
  • Technique Classification: This subtask showed lower performance, with an F1-score of 0.173 for the full 36 classes. This is attributed to class imbalance and the complexity of the task.
  • Meta-Class Improvement: Combining labels into six broader meta-classes significantly improved performance, with the model achieving an F1-score of 0.42 (and 0.44 with meta-labels). This suggests that grouping similar techniques can mitigate issues with rare classes.
  • Whole Text Classification: A Random Forest model achieved 0.9 accuracy and a macro-averaged F1-score of 0.4 on the test sample.

Limitations and Future Work

  • Dataset Limitations: The ZenPropaganda dataset is limited to Russian-language COVID-19 texts from specific platforms, potentially limiting generalizability. The current state-of-the-art methods show low performance on this dataset, indicating they are not yet fully effective for complete automated detection.
  • Future Work:
    • Expand the list of manipulative techniques and explore different grouping schemes.
  • Label texts from other popular Russian domains.
  • Investigate few-shot learning techniques, potentially using larger language models like GPT-3.5-Turbo.
  • Further research into the theoretical understanding of current manipulative media tools is needed.

Summary updated

BibTeX
@inproceedings{chernyavskiy2024zenpropaganda,
  author = {Anton Chernyavskiy and Svetlana Shomova and Irina Dushakova and Ilya Kiriya and Dmitry Ilvovsky},
  title = {{ZenPropaganda: A Comprehensive Study on Identifying Propaganda Techniques in Russian Coronavirus-Related Media}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure