Quantify the Political Bias in News Edits: Experiments with Few-Shot Learners (Student Abstract)

Preetika Verma, Hansin Ahuja, Kokil JaidkaAAAI 2023 · DOI

Abstract

News headlines are not static; they undergo a lifecycle of revisions that can subtly shift geopolitical narratives. This study investigates "dynamic news framing" by classifying headline edits into objective and subjective categories using few-shot learning with GPT-3. Analyzing a subset of the NewsEdits dataset focusing on major outlets and specific countries of interest, we find that while automated systems show promise in detecting subjective additions, they struggle significantly with identifying the intent behind content removals. This suggests that editorial "silence" remains a challenging and subtle form of media bias to quantify.

Introduction

Traditional media studies often analyze news framing through static snapshots of articles. However, modern digital journalism allows for continuous revisions, from minor typo corrections to significant shifts in narrative focus. These edits serve as a tool for "agenda-setting"—refocusing audience attention or removing controversial viewpoints to enforce specific narratives. This paper addresses the gap in dynamic news framing by quantifying how outlets like the New York Times and Fox News revise headlines when reporting on countries with complex U.S. relations, such as China, Iran, and Israel.

Methodology

The researchers developed a multidisciplinary framework combining content analysis with Large Language Models (LLMs):

  1. Taxonomy Development: Edits were categorized into four types: Objective Addition, Subjective Addition, Objective Removal, and Subjective Removal.
  2. Few-Shot Learning: GPT-3 was utilized in a few-shot configuration, provided with a small set of hand-annotated examples to distinguish between factual updates and subjective framing.
  3. Human Annotation: A gold-standard dataset of 200 headline pairs was created with an 87.5% inter-coder agreement.
  4. Geopolitical Filtering: Dictionary-based methods isolated headlines related to six countries: China, Afghanistan, Syria, Iran, Israel, and France.

Dataset and Scale

The study utilizes the NewsEdits dataset, which contains 1.2 million articles. For this analysis, the focus was narrowed to four outlets: Fox News, Rebel, New York Times, and Washington Post (WaPo).

Table 1: Summary of the NewsEdits dataset across four major outlets.

As shown in Table 1, the New York Times exhibited the highest volume of headline updates, with nearly 60,000 revisions, followed by the Washington Post with over 9,500. Headline updates were found to occur in approximately 17% of version pairs across the dataset.

Results and Performance

The experiments revealed a distinct hierarchy in the ease of detecting different types of edits. While additions often provide explicit context or guidance, removals are more subtle and difficult for AI to interpret.

Table 2: Performance metrics for GPT-3 few-shot classification across edit categories.

As detailed in Table 2, GPT-3 achieved its highest accuracy (0.89) for Objective Removals. However, the Minority-F1 scores for removals were notably low (0.12–0.13), indicating that the model often defaulted to the majority class and failed to capture the nuance of why information was deleted. In contrast, Subjective Additions were handled more effectively (Macro-F1 of 0.64), suggesting that LLMs are better at detecting the "flavor" of added biased language than the "void" left by removed content.

Discussion and Conclusion

The findings highlight that headline updates are a frequent but often invisible form of bias. The difficulty in classifying removals suggests that editorial subtraction is a more sophisticated form of gatekeeping that current AI models struggle to interpret.

Future work will aim to address the class imbalance in the dataset and expand the analysis to include social identity categories such as gender, race, and religion. Additionally, the researchers plan to compare LLM performance against traditional NLP methods focusing on sentiment and semantic similarity to better capture the evolution of news frames over time.

Summary updated

BibTeX
@inproceedings{verma2023quantify,
  author = {Preetika Verma and Hansin Ahuja and Kokil Jaidka},
  title = {{Quantify the Political Bias in News Edits: Experiments with Few-Shot Learners (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27037},
}
Figure