FANG-COVID: A New Large-Scale Benchmark Dataset for Fake News Detection in German

Justus Mattern, Yu Qiao, Elma Kerz, Daniel Wiechmann, Markus Strohmaier2021 · DOI

Abstract

As the world continues to fight the COVID-19 pandemic, it is simultaneously fighting an 'infodemic'—a flood of disinformation and spread of conspiracy theories leading to health threats and the division of society. To combat this infodemic, there is an urgent need for benchmark datasets that can help researchers develop and evaluate models geared towards automatic detection of disinformation. While there are increasing efforts to create adequate, open-source benchmark datasets for English, comparable resources are virtually unavailable for German, leaving research for the German language lagging significantly behind. In this paper, we introduce the new benchmark dataset FANG-COVID consisting of 28,056 real and 13,186 fake German news articles related to the COVID-19 pandemic as well as data on their propagation on Twitter. Furthermore, we propose an explainable textual-and social context-based model for fake news detection, compare its performance to "blackbox" models and perform feature ablation to assess the relative importance of human-interpretable features in distinguishing fake news from authentic news.

Introduction: The Infodemic Challenge

The COVID-19 pandemic was accompanied by a secondary crisis: an "infodemic" of disinformation. While the Natural Language Processing (NLP) community rapidly produced datasets to track and combat this in English, the German-speaking world lacked large-scale, localized benchmarks. Existing German datasets were often too small for modern deep learning or lacked "social context"—the metadata regarding how news is shared. FANG-COVID addresses this gap, providing a robust foundation for German fake news detection.

The FANG-COVID Dataset

The primary contribution of this work is the FANG-COVID dataset, which significantly outperforms previous German benchmarks in scale and richness. It contains 41,242 articles labeled via distant supervision based on publisher reputation. Crucially, it includes 363,393 associated tweets from over 52,000 users, allowing researchers to model the social propagation of disinformation alongside textual content.

Comparison of FANG-COVID with existing COVID-19 fake news datasets, highlighting its superior size and inclusion of social features.

Methodology: Balancing Performance and Interpretability

The researchers developed a dual-branch model architecture designed to compare high-performance "black-box" approaches with human-interpretable features.

The proposed model architecture combining interpretable textual features (CoCoGen) and social context representation.
  1. Interpretable Representation: This branch utilizes TF-IDF and CoCoGen, a tool that extracts 118 linguistic features measuring "complexity contours"—the shifts in syntactic and lexical sophistication throughout an article.
  2. Social Context Representation: This branch models the "social fingerprint" of the news using a Bidirectional LSTM (Bi-LSTM) to process Twitter sequences, including user profile metrics (follower counts, account age) and engagement statistics.
  3. Black-Box Baseline: A fine-tuned German BERT model serves as the state-of-the-art benchmark for comparison.

Key Insights: The Complexity Paradox

A surprising finding was the "Complexity Paradox": in this German dataset, fake news often exhibited higher syntactic and lexical complexity than real news. This contradicts many English-language studies where fake news is typically simpler. In the German context, disinformation often adopts a "pseudo-scientific" or overly formal tone to project authority.

Furthermore, feature ablation revealed that Twitter profile data was the most significant predictor of veracity. Accounts sharing authentic news typically had higher follower counts and fewer "friends," suggesting institutional or reputable sources.

Results and Performance

The models were evaluated on standard and "publisher-separated" splits to test generalization.

  • Top Performance: The BERT + Social Context model achieved an F1-score of 0.966 on standard splits.
  • Explainable Success: The Interpretable + Social Context model reached an F1-score of 0.900, demonstrating that social metadata can effectively bridge the gap between simple feature-based models and massive transformers.
  • Generalization Gap: Performance dropped significantly on the publisher-separated split (BERT F1 ~0.68), highlighting the difficulty models face in distinguishing general "fakeness" from the specific writing styles of individual outlets.

Conclusion

FANG-COVID provides a vital resource for the German NLP community. While transformer models like BERT offer the highest accuracy, the study proves that incorporating social context and linguistic complexity can yield high-performing, explainable models. Future work will focus on improving generalization and exploring more complex network propagation patterns to stay ahead of evolving disinformation tactics.

Summary updated

Abstract

As the world continues to fight the COVID-19 pandemic, it is simultaneously fighting an 'infodemic' -a flood of disinformation and spread of conspiracy theories leading to health threats and the division of society. To combat this infodemic, there is an urgent need for benchmark datasets that can help researchers develop and evaluate models geared towards automatic detection of disinformation. While there are increasing efforts to create adequate, open-source benchmark datasets for English, comparable resources are virtually unavailable for German, leaving research for the German language lagging significantly behind. In this paper, we introduce the new benchmark dataset FANG-COVID consisting of 28,056 real and 13,186 fake German news articles related to the COVID-19 pandemic as well as data on their propagation on Twitter. Furthermore, we propose an explainable textual-and social context-based model for fake news detection, compare its performance to "blackbox" models and perform feature ablation to assess the relative importance of humaninterpretable features in distinguishing fake news from authentic news.
BibTeX
@misc{mattern2021fang,
  author = {Justus Mattern and Yu Qiao and Elma Kerz and Daniel Wiechmann and Markus Strohmaier},
  title = {{FANG-COVID: A New Large-Scale Benchmark Dataset for Fake News Detection in German}},
  year = {2021},
  doi = {10.18653/v1/2021.fever-1.9},
}
Figure