Pseudo-labelling Enhanced Media Bias Detection

Qin Ruan, Brian Mac Namee, Ruihai DongarXiv 2021 · arXiv

Introduction

Detecting media bias—specifically "hyperpartisan" reporting where a story is presented through a heavily one-sided lens—is a significant challenge for machine learning. To train these models, researchers typically rely on two types of data: manual annotation and distant supervision. While manual annotation is highly accurate, it is slow and expensive. Conversely, distant supervision (labeling articles based on a publisher’s reputation) is fast and generates massive datasets but is inherently "noisy," as neutral reports from partisan outlets are incorrectly labeled as biased.

This paper introduces a simple yet powerful "overlap-checking" mechanism to bridge this gap. By using a model trained on a small set of high-quality manual labels to filter a massive, noisy dataset, the researchers created a high-performing hybrid training set that outperformed previous state-of-the-art models on the SemEval-2019 hyperpartisan news detection task.

The Overlap-Checking Mechanism

The authors propose a three-step process to leverage the scale of noisy data without inheriting its errors:

  1. Initial Training: A neural network (utilizing an ELMo-based sentence encoder and CNN layers) is trained on a small, clean, manually labeled dataset until convergence.
  2. Overlap Selection: This "well-trained" model predicts labels for the massive noisy dataset. The system identifies the "overlap"—samples where the model’s prediction matches the noisy "by-publisher" label. The logic is that if a model trained on clean data agrees with a noisy label, that sample is likely a high-quality example of bias.
  3. Joint Re-training: The top $N$ samples from this overlap, ranked by confidence, are combined with the original clean data to re-train the network from scratch.
The three-step overlap-checking mechanism for generating high-quality pseudo-labeled data.

This approach effectively acts as a filter, distilling the massive noisy dataset into a smaller, higher-quality subset that reinforces the patterns learned from expert annotations.

Experimental Setup and Datasets

The study utilized the dataset from Task 4 of the 2019 SemEval challenge, consisting of:

  • Clean Set: 1,273 manually labeled instances.
  • Noisy Set: 754,000 instances annotated via distant supervision (by-publisher).

The model, named "Otto Cheirk," was tested against the official competition leaderboard to evaluate its effectiveness in hyperpartisan news detection.

Key Results and Insights

The core innovation of this work is using consistency between a weak supervisor (the publisher label) and a strong but data-constrained model as a reliable proxy for ground truth. This allows the model to overcome the "noise floor" that often causes models trained on massive noisy datasets to perform worse than those trained on small clean ones.

In head-to-head comparisons with the top four teams from the original SemEval competition, the Otto Cheirk model achieved superior results in Accuracy, Recall, and F1 score.

Performance comparison of the Otto Cheirk model against top submissions in the SemEval-2019 task.

While the model saw a slight dip in precision compared to the previous best (Bertha von Suttner), the significant jump in Recall (from 0.755 to 0.844) indicates that the pseudo-labeling helped the model identify many more biased articles that baseline models were missing.

Conclusion and Future Work

This paper provides a concise proof-of-concept for a data-centric approach to bias detection. The mechanism is architecture-agnostic and can be applied to almost any text classification pipeline. Future research may explore the optimal volume of added data ($N$), the application of this method to more granular labels (such as specific political leanings), and methods to reduce the computational cost of the multiple training passes required.

Summary updated

Abstract

Leveraging unlabelled data through weak or distant supervision is a compelling approach to developing more effective text classification models. This paper proposes a simple but effective data augmentation method, which leverages the idea of pseudo-labelling to select samples from noisy distant supervision annotation datasets. The result shows that the proposed method improves the accuracy of biased news detection models.
BibTeX
@misc{ruan2021pseudo,
  author = {Qin Ruan and Brian Mac Namee and Ruihai Dong},
  title = {{Pseudo-labelling Enhanced Media Bias Detection}},
  year = {2021},
  eprint = {2107.07705},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.07705},
}
Figure