Abstract
In the Russo-Ukrainian war, propaganda is produced by Russian state-run news outlets for both international and domestic audiences. Its content and form evolve and change with time as the war continues. This constitutes a challenge to content moderation tools based on machine learning when the data used for training and the current news start to differ significantly. In this follow-up study, we evaluate our previous BERT and SVM models that classify Pro-Kremlin propaganda from a Pro-Western stance, trained on the data from news articles and telegram posts at the start of 2022, on the new 2023 subset. We examine both classifiers’ errors and perform a comparative analysis of these subsets to investigate which changes in narratives provoke drops in performance.
Introduction
The Russo-Ukrainian war has highlighted the dynamic nature of state-sponsored propaganda. As military objectives and political landscapes shift, so do the narratives used to justify them. This phenomenon, known in machine learning as "concept drift," poses a significant challenge for automated content moderation. Models trained on early-war data may struggle to identify later propaganda as the linguistic "fingerprints"—keywords, emotional tones, and structural features—evolve. This study investigates the robustness of existing detection models against the propaganda landscape of 2023.
Methodology
We compared two distinct classification architectures: a Support Vector Machine (SVM) utilizing 41 handcrafted linguistic features and 116 keywords, and a Multilingual BERT (mBERT) model leveraging contextual embeddings. To analyze the evolution of propaganda, we employed the Mann-Whitney U-test to compare feature distributions between 2022 and 2023 datasets. Error analysis was conducted to identify why models misclassified certain texts, focusing on feature importance for SVM and token attribution for BERT.
Datasets
The analysis utilized two primary datasets:
- 2022 Subset: Approximately 18,000 texts collected between February and April 2022.
- 2023 Subset: 1,400 texts collected in January 2023. Data spanned Russian, Ukrainian, Romanian, French, and English, sourced from both traditional news outlets and Telegram channels. Labels were assigned based on source reputation rather than individual content review.
Narrative Evolution and Feature Analysis
Our analysis revealed significant shifts in propaganda tactics. The initial branding of the "Special Military Operation" has seen a decline in frequency within state-run outlets. In contrast, Russian Telegram channels have increasingly adopted aggressive terminology, such as "Kiev Regime," and high-modality language. Interestingly, the word "war"—initially a marker of Pro-Western stances due to Russian legal restrictions—has transitioned into a predictor for Pro-Kremlin propaganda in certain contexts, such as discussions regarding "war tribunals."
Model Performance and Robustness
Despite the narrative shifts, both models demonstrated remarkable resilience. BERT achieved an F1-score of 0.93 on the 2023 data, actually outperforming its 2022 test results. The SVM model remained stable with an F1-score of 0.85. However, both models struggled with "liberal" Russian news sources, which often share linguistic similarities with state media despite opposing stances. Additionally, the SVM model faced difficulties with the informal style of Ukrainian Telegram posts, resulting in a significant misclassification rate for that specific platform.
Conclusion
While automated models remain effective at identifying propaganda over time, the reliance on source-based labeling remains a limitation, as models may learn outlet-specific styles rather than general propaganda characteristics. Future efforts will focus on expanding language coverage and integrating expert fact-checking to refine labeling and improve model transparency.
Summary updated