Fine-grained Classification of Political Bias in German News: A Data Set and Initial Experiments

Dmitrii Aksenov, Peter Bourgonje, Karolina Zaczynska, Malte Ostendorff, Julian Moreno-Schneider, Georg RehmACL 2021

TL;DR

This paper introduces a novel, fine-grained dataset of German news articles labeled for political bias, moving beyond simple hyperpartisan detection. They evaluate various classification models, finding that while performance drops compared to binary classification, this nuanced approach is crucial for understanding online abuse and hate speech.

What Problem Does This Paper Address?

The internet has democratized information dissemination, but this also brings challenges in discerning information quality. Detecting political bias in news is crucial for understanding online abuse, hate speech, and for general media literacy. While previous work has focused on binary classification (hyperpartisan vs. not) and primarily on English-language news, this paper argues for a more nuanced, five-point scale (far-left to far-right) and extends this research to the German language. Understanding the direction of bias, not just its presence, is vital for transparency and credibility.

Methods

The authors experiment with several classification approaches:

  • Bag-of-Words (BOW): A classic text representation where each word is a feature.
  • TF-IDF: Similar to BOW but weights words based on their importance in a document relative to the corpus.
  • BERT: Leveraging pre-trained transformer models (specifically, a German BERT base model) for contextualized word embeddings.

These features are then fed into various classification models:

  • Logistic Regression: A standard linear model.
  • Naive Bayes: A probabilistic classifier based on conditional independence.
  • Random Forest: An ensemble of decision trees, with a weighted version to handle class imbalance.
  • EasyEnsemble: Another ensemble method specifically designed to address class imbalance by creating balanced training sets.

Datasets

The paper introduces a new dataset of German news articles. This dataset was created using a semi-supervised approach:

  • Publisher-based labeling: They utilized a survey from Medienkompass.org, which rated 40 German media outlets on partiality. These ratings were converted into a 5-point political bias scale (far-left, center-left, center, center-right, far-right).
  • Data Collection: News articles were collected through two crawling methods, one using the news-please crawler.
  • Data Cleaning: The collected data underwent rigorous cleaning, including duplicate removal, filtering of non-news articles, and removal of texts published before 2018 to mitigate data sparsity issues.

For benchmarking, they also used the data from the 2019 Hyperpartisan News Detection task, which features binary labels (hyperpartisan/not hyperpartisan) for English news articles.

Key Insights and Ideas

  • Fine-grained bias is crucial: Moving beyond a simple "hyperpartisan" label to a spectrum (far-left to far-right) is essential for a deeper understanding of online content and its potential for abuse.
  • German language is under-represented: There's a clear need for more research and datasets in languages other than English to address global issues of online bias.
  • Class imbalance is a major hurdle: The distribution of political bias in news articles is uneven, posing a significant challenge for classification models, especially for the "far-left" and "far-right" categories.
  • Publisher identity can be a strong signal (and a confounder): While publisher identity is useful for labeling, models can easily overfit to it, making it crucial to ensure disjoint publisher sets between training and testing.

Key Results

  • Performance drop with fine-grained labels: When moving from binary classification (achieving 86% accuracy on a binary task) to the 5-class political bias task, the accuracy dropped significantly to 43% for the best-performing setup (Naive Bayes + BOW). This highlights the difficulty of the fine-grained task.
  • BOW + Naive Bayes and TF-IDF + Random Forest excel: These traditional methods, especially when combined with careful handling of class imbalance, performed best on their German dataset.
  • BERT's performance is surprisingly low: Despite its success in many NLP tasks, BERT-based models did not outperform simpler methods on this specific task, even when using a German-specific BERT model. This might be due to the nature of the data or the specific fine-tuning strategy.
  • Topical insights: Approximately 40% of the collected articles were related to COVID-19, reflecting the temporal scope of the data collection.

Limitations and Future Work

  • Publisher-based labeling limitations: While efficient, this method might not capture the nuances of individual article bias, especially if a publisher's stance shifts or if articles within a publication vary in bias. The authors suggest that large-scale, individual article labeling is a crucial future direction.
  • Data volume for minority classes: The small number of articles and publishers for the "far-left" class presented a significant challenge.
  • BERT performance: The authors acknowledge the surprisingly low performance of BERT and suggest further investigation into its impact, potentially exploring different fine-tuning strategies or model architectures.
  • Quality vs. Bias: The paper focuses on bias and leaves the classification of media quality for future research.
  • Cross-lingual and cross-domain generalization: Further research is needed to understand how models trained on one language or domain perform on others.

Summary updated

BibTeX
@inproceedings{aksenov2021fine,
  author = {Dmitrii Aksenov and Peter Bourgonje and Karolina Zaczynska and Malte Ostendorff and Julian Moreno-Schneider and Georg Rehm},
  title = {{Fine-grained Classification of Political Bias in German News: A Data Set and Initial Experiments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure