TL;DR
This paper introduces "ZenPropaganda," a novel, meticulously annotated dataset of Russian-language COVID-19-related texts from VKontakte and Yandex.Zen, designed to identify propaganda techniques. The authors also propose a new multi-level classification scheme and evaluate state-of-the-art models, highlighting the challenges and nuances of detecting propaganda in real-world social media content.
What Problem Does This Paper Address?
The proliferation of "fake news" and manipulative content online, especially amplified by social media and AI, poses a significant challenge to media literacy and informed public discourse. While propaganda detection is an active research area, there's a persistent need for more comprehensive datasets and refined classification schemes, particularly for non-English languages and specific domains like social media. Existing approaches often struggle with the fine-grained nature of propaganda techniques and the operationalization of these concepts for automated analysis. This paper addresses the gap by creating a specialized Russian dataset and a structured classification system to better understand and detect propaganda in a popular online context.
Methods
The core methodological contribution is the development of a novel multi-level classification scheme for propaganda techniques. This scheme categorizes techniques into three types:
- Narrative techniques: Affecting the entire text, often harder to systematize and automate.
- Fragment-level techniques: Targeting specific parts of the text.
- Markers of manipulation: Surface-level indicators (e.g., excessive capitalization, exclamation marks) that can be automatically highlighted.
The authors also adapted and evaluated Transformer-based models, specifically RoBERTa, for two subtasks:
- Span Identification (SI): Identifying the exact character spans within a text that contain propaganda. This involved using a BIO encoding scheme and incorporating a Conditional Random Field (CRF) layer for improved sequence labeling.
- Technique Classification (TC): Assigning one or more propaganda technique labels to identified spans.
Datasets
The primary contribution is the ZenPropaganda dataset.
- Source: Texts collected from VKontakte (user-generated content) and Yandex.Zen (blogging platform).
- Content: COVID-19-related texts.
- Annotation: Manually annotated with fine-grained labeling of manipulative spans.
- Size: 125 texts, containing approximately 2400 manipulative segments.
- Key Features: Texts are at least 1000 characters long, and annotations are character-level precise.
The authors also compared their dataset and findings with existing benchmarks, including:
- SemEval-2020 Task 11: A dataset for propaganda detection in news articles. ZenPropaganda differs in its focus on social media and longer manipulative spans.
- SemEval-2023 (multilingual): Another relevant dataset, though the authors note differences in subtasks and domains.
Key Insights and Ideas
- Social Media Nuances: Propaganda on platforms like VKontakte and Yandex.Zen exhibits unique characteristics, including longer manipulative spans and a tendency for authors to repeatedly use a few core techniques.
- Operationalization Challenges: Defining and consistently annotating propaganda techniques is difficult due to interpretive and terminological ambiguities, especially in Russian research.
- Intra-text Focus: The research deliberately focuses on manipulation within texts, excluding external fact-checking, which simplifies the task but limits the scope.
- Conspiracy Theories: A notable observation was the frequent use of conspiracy theories, suggesting this could be a valuable addition to future classification schemes.
- Model Performance Gap: State-of-the-art models, while adapted, show significantly lower performance on this Russian social media dataset compared to English news datasets, highlighting the need for domain-specific and language-specific research.
Key Results
- Dataset Creation: The ZenPropaganda dataset, with its fine-grained, character-level annotations, provides a valuable resource for Russian propaganda detection research.
- Span Identification: The RoBERTa-CRF model achieved an F1-score of 0.30 for span identification on the test sample, which is lower than English benchmarks but considered reasonable given the dataset's specifics (longer spans, different classes). Sentence-level F1 was 0.43.
- Technique Classification: This subtask showed lower performance, with an F1-score of 0.173 for the full 36 classes. This is attributed to class imbalance and the complexity of the task.
- Meta-Class Improvement: Combining labels into six broader meta-classes significantly improved performance, with the model achieving an F1-score of 0.42 (and 0.44 with meta-labels). This suggests that grouping similar techniques can mitigate issues with rare classes.
- Whole Text Classification: A Random Forest model achieved 0.9 accuracy and a macro-averaged F1-score of 0.4 on the test sample.
Limitations and Future Work
- Dataset Limitations: The ZenPropaganda dataset is limited to Russian-language COVID-19 texts from specific platforms, potentially limiting generalizability. The current state-of-the-art methods show low performance on this dataset, indicating they are not yet fully effective for complete automated detection.
- Future Work:
- Expand the list of manipulative techniques and explore different grouping schemes.
- Label texts from other popular Russian domains.
- Investigate few-shot learning techniques, potentially using larger language models like GPT-3.5-Turbo.
- Further research into the theoretical understanding of current manipulative media tools is needed.
Summary updated