Introduction
Identifying political bias in news articles is a critical task for combating "filter bubbles" and polarization. However, training machine learning models for this task faces a significant bottleneck: data labeling. Manually annotating thousands of long-form articles for political slant is time-consuming and expensive.
In contrast, short-form content like tweets is easy to label at scale by observing the political affiliation of the account. However, models trained on tweets often perform poorly on long articles due to opinion concentration. While a tweet is usually a concentrated burst of opinion, a news article contains many apolitical, factual, or "filler" sentences that act as noise, confusing models that expect a high density of polarized language. KnowBias addresses this by leveraging easy-to-label Twitter data and implementing a two-step pipeline that filters out neutral sentences before classification.
Methodology
The authors propose a two-step classification scheme designed for domain adaptation, moving from short-form training data to long-form inference. The system utilizes Universal Sentence Encoders (USE) to convert text into high-dimensional vectors, ensuring semantic meaning is preserved regardless of specific word choices.
The pipeline consists of the following stages:
- Tokenizer: The long document is broken down into individual sentences.
- Neutral Detector: A deep neural network (DNN) distinguishes between "polarized" and "neutral" text. This model is trained on a dataset of political tweets versus a general stream of random tweets.
- Sentence Fusion: Sentences deemed apolitical by the Neutral Detector are discarded, and the remaining polarized sentences are fused back together to create a condensed version of the text.
- Polarity Classifier: A separate DNN, trained on political tweets to classify them on a left-to-right spectrum, predicts the final polarity of the condensed text.
Datasets
The study utilized several distinct datasets for training and evaluation:
- Polarity Training Data: Approximately 150,000 tweets from 28 verified politicians and media personalities.
- Neutrality Training Data: Roughly 80,000 neutral tweets sampled from the general Twitter stream.
- Evaluation Data: A split of 20,000 tweets and 24 articles from mainstream news outlets. Ground truth for the articles was established via crowdsourced annotations from 79 respondents.
Key Results and Insights
The two-step approach showed a marked improvement over the baseline (one-step) approach when applied to long-form articles.
The "Dilution" Effect
The core insight is that the primary difference between a tweet and an article is the density of the signal. In long-form text, neutral sentences "dilute" the political slant. The authors proved this by artificially injecting neutral sentences into polarized data; while a standard classifier's accuracy plummeted as noise increased, the two-step method maintained higher accuracy by filtering the noise.
Performance Metrics
- Accuracy Boost: On long-form articles, accuracy increased from 66.67% (one-step) to 75.00% (two-step).
- Consistency: Using Spearman-rho rank correlation to match human "crowd" opinions, the two-step system achieved a correlation of 0.69, significantly higher than the baseline's 0.52.
- Neutral Detection: The neutral detector itself was highly effective, achieving 95.63% accuracy in identifying apolitical vs. polarized tweets.
Discussion and Future Work
The paper demonstrates that authorship on social media can serve as a robust proxy to bypass expensive manual labeling for complex documents. However, several challenges remain for future development:
- Temporal Shifts: Political language evolves; models may require continuous updates to remain relevant to the current landscape.
- Contextual Nuance: Training the neutral detector on news-specific neutral sentences, rather than random tweets, could better handle the nuances of journalistic writing.
- Cohesion: The impact of breaking contextual flow by filtering sentences on sentence embeddings requires further exploration to ensure semantic integrity is fully maintained.
Summary updated