POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection

Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, Lu WangNAACL 2022 · DOI

Introduction

Identifying political ideology and stance in text remains a significant challenge for natural language processing. Unlike topical classification, ideology is often conveyed through subtle "framing"—the selective inclusion of facts, the highlighting of specific entities, and the sentiment attached to them. While large language models (LLMs) excel at general language tasks, they frequently struggle with these nuances, often relying on "shortcuts" like news-source-specific boilerplate rather than underlying ideological perspectives.

To address these limitations, researchers have developed POLITICS, a language model specifically pretrained to understand political ideology. By leveraging a massive dataset of "same-story" news clusters, the model learns to identify bias by comparing how different media outlets (e.g., Daily Kos vs. Breitbart) frame the exact same event.

Methodology: Comparative Pretraining

The POLITICS model is built upon a RoBERTa architecture, undergoing "continued pretraining" with three specialized objectives designed to capture ideological nuance:

  1. Ideology-Driven Triplet Loss: This objective uses a triplet of articles: an Anchor (e.g., a CNN report on a specific bill), a Positive sample (another Left-leaning article on the same bill), and a Negative sample (a Right-leaning article on the same bill). The model is trained to minimize the distance between the Anchor and the Positive sample in vector space while maximizing the distance from the Negative sample.
  2. Story-Driven Triplet Loss: To prevent the model from simply memorizing the stylistic "voice" of a specific outlet, the researchers introduced a story-based objective. Here, the negative sample is an article from the same outlet but covering a different story. This forces the model to focus on the content of the story rather than the outlet's boilerplate language.
  3. Entity- and Sentiment-aware Masked Language Modeling (MLM): Standard MLM hides random tokens. POLITICS specifically targets Named Entities (e.g., politicians, organizations) and Sentiment words. Because political stance is often defined by how specific actors are characterized, this objective helps the model identify who is being praised or blamed in a given narrative.

Data Resources: BIGNEWS and BIGNEWSALIGN

The study introduces two substantial resources for the political NLP community:

  • BIGNEWS: A comprehensive collection of 3.6 million US political news articles from 11 outlets across the ideological spectrum, spanning the years 2000 to 2021.
  • BIGNEWSALIGN: A subset of 1.06 million articles where stories from different outlets are "aligned" because they cover the same event, providing the basis for the comparative pretraining objectives.

Experimental Results

The model was evaluated on 11 distinct tasks across 8 datasets, including diverse domains such as Congressional speeches, YouTube comments, and Twitter data.

State-of-the-Art Performance

POLITICS outperformed strong baselines, including standard RoBERTa and SVMs, on 8 out of 11 tasks. Notably, it achieved an F1 score of 77.0 on the VAST stance detection benchmark, a significant improvement over the previous state-of-the-art of 69.2.

Robustness and Generalization

The model demonstrated a 10% improvement over RoBERTa on long-form documents and formal texts, where ideological framing is typically more subtle. Furthermore, POLITICS showed remarkable "few-shot" robustness; even with as few as 16 labeled examples, the model maintained a significant lead over its predecessors, suggesting a strong innate understanding of political language.

Key Insights

The researchers' analysis yielded several fascinating insights into the nature of political language:

  • Relativity of Ideology: The success of the model suggests that ideology is best understood relatively. Bias is most visible not in isolation, but when comparing different versions of the same story.
  • Republican Coherence: A side analysis revealed that the model had lower "perplexity" (was less surprised) by Right-leaning and Center articles compared to Left-leaning ones. This supports political science theories suggesting that the Republican media ecosystem in the US is more linguistically unified than the Democratic one.
  • Salient Entity Focus: Visualization of attention heads showed that POLITICS successfully learned to focus on politically salient entities (e.g., "Trump") and their surrounding descriptors, whereas standard models exhibited more scattered attention patterns.

Limitations and Future Directions

While POLITICS represents a significant step forward, several challenges remain. The training data is primarily text-based journalism, leaving a gap in understanding ideology in TV transcripts, images, or video. Additionally, while the model generalizes better than previous iterations, the transition from formal news to informal social media "slang" still results in a performance drop. Finally, the authors caution that such models should be used as tools to assist human researchers rather than as definitive arbiters of "truth."

Summary updated

BibTeX
@inproceedings{liu2022politics,
  author = {Yujian Liu and Xinliang Frederick Zhang and David Wegsman and Nicholas Beauchamp and Lu Wang},
  title = {{POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.101},
}
Figure