TL;DR
This paper introduces a novel pre-training framework that leverages social and linguistic cues to improve political perspective identification in news articles. By incorporating knowledge about political entities, social sharing patterns, and framing indicators, their method significantly enhances the performance of text models, especially in low-data scenarios.
What Problem Does This Paper Address?
Understanding the political perspective behind news reporting is crucial in today's polarized media landscape. This helps prevent "echo chambers" and ensures a more balanced representation of viewpoints. However, current deep learning models for political perspective identification often require extensive labeled data, which is costly to obtain. Furthermore, general-purpose pre-trained models like BERT, while powerful, struggle to capture the subtle linguistic and contextual nuances that signal bias. This paper aims to bridge this gap by developing a pre-training strategy that imbues text models with a richer understanding of political bias through readily available social and linguistic context, making them more effective and data-efficient for downstream tasks.
Methods
The core of this work is a novel pre-training framework designed to adapt existing text models for political perspective identification. The framework introduces three self-supervised pre-training objectives:
- Entity-Guided Pre-training: This objective injects knowledge about political entities (people, organizations) into the model. It uses Wikipedia2Vec to learn entity embeddings and then employs an entity-level masking task, similar to BERT's masked language modeling, to predict masked entities based on their context.
- Sharing User-Guided Pre-training: This objective leverages the intuition that users with similar political leanings tend to share similar news articles. It uses a social information graph (users, articles, political figures) and Graph Convolutional Networks (GCNs) to learn user embeddings. The pre-training task then involves predicting the sharing user given a news article, encouraging the model to learn representations that capture article bias based on user behavior.
- Frame Indicator-Guided Pre-training: This objective focuses on political framing, a strategy used to influence perception without altering facts. It identifies "frame indicators" – words strongly associated with specific frames – and uses them in a masked token prediction task to help the model distinguish between different framing strategies.
These pre-trained models are then integrated into a Hierarchical Multi-Head Attention Network (MAN) for the final political perspective identification task. The MAN architecture uses bidirectional LSTMs for word and sentence encoding, followed by attention mechanisms to capture important information at different levels.
Datasets
The authors evaluate their framework on two publicly available datasets:
- Allsides: This dataset consists of news articles labeled with political perspectives (left, center, right). It was collected from news aggregation websites and uses crowdsourced and editorial reviews for bias labeling.
- SemEval: This is the official training dataset from SemEval 2019 Task 4: Hyperpartisan News Detection. It provides a binary label (hyperpartisan or not) for news articles.
The paper does not introduce new datasets but uses existing ones for rigorous evaluation and comparison.
Key Insights and Ideas
- Pre-training is Key for Nuance: The research strongly suggests that pre-training on social and linguistic context is crucial for capturing the subtle forms of bias in news articles, which general-purpose models often miss.
- Complementarity of Signals: The ensemble of models trained with different pre-training objectives (entity, sharing, frame) significantly outperforms individual models, highlighting that these diverse sources of information are complementary.
- Data Efficiency: The pre-training strategy demonstrates a substantial performance gain, particularly when labeled training data is scarce. This is a critical advantage for real-world applications where obtaining large labeled datasets is challenging.
- Social Signals as Noisy Labels: The sharing-guided pre-training, while not always outperforming others with abundant data, shows remarkable performance with limited supervision. This suggests that social sharing patterns, even if noisy, can serve as valuable weak supervision signals.
Key Results
- Significant Performance Gains: The pre-trained MAN models consistently outperformed baseline models (including BERT) on both the Allsides and SemEval datasets. For instance, on the Allsides dataset, the ensemble model achieved an accuracy of 83.74% and a macro F1 score of 82.84%, compared to BERT's 81.55% accuracy and 80.13% macro F1.
- Contribution of Each Signal: Ablation studies showed that removing any of the pre-training objectives (entity, sharing, frame) led to a performance drop, confirming the individual contributions of each signal.
- Improved Bias Span Identification: In qualitative results on the BASIL dataset, the entity-guided pre-trained model showed higher attention scores assigned to human-annotated bias spans, indicating better localization of biased content.
- Effectiveness with Limited Data: Figure 2 illustrates that as the amount of training data decreases, the performance gap between the pre-trained and non-pre-trained models widens, with the sharing-guided pre-training showing particularly strong gains in low-data regimes.
Limitations and Future Work
- Interdependence of Contexts: The authors acknowledge that the social and linguistic contexts they exploit are not independent and suggest extending their work to pre-train better text models by incorporating information from various sources together more deeply.
- Data Availability for Social Graph: The sharing user-guided pre-training was not evaluated on the SemEval dataset due to the lack of available social graph information for that dataset.
- Ensemble Complexity: While the ensemble approach yields the best results, it increases model complexity. Future work could explore more integrated ways to combine these signals within a single model.
Summary updated