Introduction
Detecting media bias—specifically "hyperpartisan" reporting where a story is presented through a heavily one-sided lens—is a significant challenge for machine learning. To train these models, researchers typically rely on two types of data: manual annotation and distant supervision. While manual annotation is highly accurate, it is slow and expensive. Conversely, distant supervision (labeling articles based on a publisher’s reputation) is fast and generates massive datasets but is inherently "noisy," as neutral reports from partisan outlets are incorrectly labeled as biased.
This paper introduces a simple yet powerful "overlap-checking" mechanism to bridge this gap. By using a model trained on a small set of high-quality manual labels to filter a massive, noisy dataset, the researchers created a high-performing hybrid training set that outperformed previous state-of-the-art models on the SemEval-2019 hyperpartisan news detection task.
The Overlap-Checking Mechanism
The authors propose a three-step process to leverage the scale of noisy data without inheriting its errors:
- Initial Training: A neural network (utilizing an ELMo-based sentence encoder and CNN layers) is trained on a small, clean, manually labeled dataset until convergence.
- Overlap Selection: This "well-trained" model predicts labels for the massive noisy dataset. The system identifies the "overlap"—samples where the model’s prediction matches the noisy "by-publisher" label. The logic is that if a model trained on clean data agrees with a noisy label, that sample is likely a high-quality example of bias.
- Joint Re-training: The top $N$ samples from this overlap, ranked by confidence, are combined with the original clean data to re-train the network from scratch.
This approach effectively acts as a filter, distilling the massive noisy dataset into a smaller, higher-quality subset that reinforces the patterns learned from expert annotations.
Experimental Setup and Datasets
The study utilized the dataset from Task 4 of the 2019 SemEval challenge, consisting of:
- Clean Set: 1,273 manually labeled instances.
- Noisy Set: 754,000 instances annotated via distant supervision (by-publisher).
The model, named "Otto Cheirk," was tested against the official competition leaderboard to evaluate its effectiveness in hyperpartisan news detection.
Key Results and Insights
The core innovation of this work is using consistency between a weak supervisor (the publisher label) and a strong but data-constrained model as a reliable proxy for ground truth. This allows the model to overcome the "noise floor" that often causes models trained on massive noisy datasets to perform worse than those trained on small clean ones.
In head-to-head comparisons with the top four teams from the original SemEval competition, the Otto Cheirk model achieved superior results in Accuracy, Recall, and F1 score.
While the model saw a slight dip in precision compared to the previous best (Bertha von Suttner), the significant jump in Recall (from 0.755 to 0.844) indicates that the pseudo-labeling helped the model identify many more biased articles that baseline models were missing.
Conclusion and Future Work
This paper provides a concise proof-of-concept for a data-centric approach to bias detection. The mechanism is architecture-agnostic and can be applied to almost any text classification pipeline. Future research may explore the optimal volume of added data ($N$), the application of this method to more granular labels (such as specific political leanings), and methods to reduce the computational cost of the multiple training passes required.
Summary updated