COVID-19 in Bulgarian Social Media: Factuality, Harmfulness, Propaganda, and Framing

Preslav Nakov, Firoj Alam, Shaden Shaar, Giovanni Da San Martino, Yifan ZhangRANLP 2021 · OpenReview

Abstract

This study presents the first comprehensive analysis of the COVID-19 "infodemic" within Bulgarian social media. By examining disinformation, propaganda, and framing in a low-resource language context, the researchers combine manual annotation of 4,000 tweets with automated analysis. The findings reveal that while the majority of content remains factual, "skeptical" narratives heavily utilize specific propaganda techniques—such as doubt and flag-waving—to pivot the discourse from public health concerns toward political and economic grievances.

Introduction

The COVID-19 pandemic catalyzed a global "infodemic," characterized by a volatile mix of medical misinformation and political disinformation. While extensive research has mapped these trends in high-resource languages like English and Arabic, smaller language communities, such as Bulgaria, have remained largely understudied. This research bridges that gap by analyzing the Bulgarian social media landscape through four critical dimensions: factuality, harmfulness, propaganda techniques, and thematic framing.

Methodology

The researchers implemented a hybrid pipeline that integrated human expertise with advanced Natural Language Processing (NLP) models. This multi-stage approach allowed for both granular qualitative assessment and large-scale quantitative analysis.

The system architecture illustrates the pipeline from data collection and filtering to the parallel manual and automatic analysis tracks.

Manual and Automatic Analysis

  • Manual Annotation: Three fluent Bulgarian speakers annotated a subset of 4,000 tweets using a seven-question schema to evaluate factuality and potential harm. Additionally, Facebook articles were categorized based on their stance as "skeptical" (anti-measures) or "concerned" (pro-measures).
  • Propaganda Detection: The study utilized two specialized systems: Proppy, a maximum entropy model for article-level detection, and Prta, a fragment-level classifier capable of identifying 18 distinct propaganda techniques.
  • Framing Analysis: Using the Tanbih system—a fine-tuned BERT model—the team detected 15 different narrative "frames," such as Economy, Morality, and Health & Safety.
  • Cross-Language Transfer: To utilize tools originally designed for English, the researchers employed Google Translate as a bridge for the Bulgarian text, a necessary methodology for low-resource language research.

Datasets

The study drew from diverse social media sources collected throughout 2020:

  1. Twitter: A corpus of 30,000 Bulgarian tweets, filtered to 4,000 for manual annotation.
  2. Facebook: Articles from partisan groups categorized into "Concerned," "Skeptical," and "Conspiracy" clusters.
  3. Benchmarks: The analysis was grounded in existing schemas from the Media Frames Corpus and prior work on COVID-19 disinformation.

Key Results and Insights

The "Skeptic" Playbook and Framing

A primary finding was the stark contrast in propaganda usage between different ideological groups. "Skeptical" articles were significantly more likely to employ techniques like doubt, flag-waving, and slogans. Conversely, "concerned" articles typically appealed to authority, fear, and emotion.

Framing also served as a tool for narrative diversion. While concerned groups focused almost exclusively on Health & Safety, skeptical groups shifted the focus toward Quality of Life, Policy, Legality, and the Economy, effectively reframing a medical crisis as a political one.

Propaganda and Harmfulness Correlation

The automated analysis demonstrated a clear link between the perceived harm of a tweet and its propagandistic content. Content identified as "check-worthy" (requiring verification) or harmful (rumors and panic-inducing content) showed a much higher likelihood of containing propaganda.

Comparison of propaganda likelihood across check-worthiness (a) and harmfulness categories (b), showing that harmful and check-worthy content correlates with higher propaganda scores.

Key statistical observations included:

  • Inter-annotator Agreement: The manual tasks achieved a Fleiss Kappa ($\kappa$) between 0.41 and 0.60, indicating moderate to substantial agreement on subjective measures like "harm."
  • Dominant Frames: In the general tweet dataset, Economy emerged as the most frequent perspective, followed by Health & Safety.
  • Panic vs. Rumor: While most Bulgarian tweets were harmless, those flagged as harmful were often specifically designed to incite panic through loaded language and exaggeration.

Limitations and Future Work

While the study provides a foundational map of the Bulgarian infodemic, the authors note several limitations. The "translation tax" incurred by using Google Translate may introduce noise or obscure subtle linguistic nuances in propaganda. Furthermore, the automated models were not natively trained on Bulgarian data. Future research aims to develop native Bulgarian NLP tools and expand the scope of analysis to other platforms like Telegram and WhatsApp to capture a more holistic view of regional disinformation trends.

Summary updated

BibTeX
@inproceedings{nakov2021covid,
  author = {Preslav Nakov and Firoj Alam and Shaden Shaar and Giovanni Da San Martino and Yifan Zhang},
  title = {{COVID-19 in Bulgarian Social Media: Factuality, Harmfulness, Propaganda, and Framing}},
  booktitle = {RANLP},
  year = {2021},
  url = {https://openreview.net/forum?id=_ufTluwVZn-},
}
Figure