When Neutral Summaries Are Not That Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries (Student Abstract)

Supriti Vijay, Aman Priyanshu, Ashiqur R. KhudaBukhshAAAI 2025 · DOI

TL;DR

This paper introduces a novel method to quantify political bias in LLM-generated news summaries, revealing a consistent liberal slant across several popular models. The findings highlight the potential for LLMs to subtly shape public opinion and underscore the need for greater transparency and bias mitigation in AI-driven content.

What Problem Does This Paper Address?

In an era where Large Language Models (LLMs) are increasingly used to summarize news and shape information consumption, understanding their political neutrality is paramount. Algorithmic curation can significantly influence societal narratives, and if LLMs exhibit biases, they could inadvertently reinforce or even manipulate public opinion, particularly concerning politically charged topics. This paper addresses the critical question of whether LLM-generated summaries, even when intended to be neutral, actually reflect a balanced political perspective, especially on divisive issues.

Methods

The authors propose a novel approach to quantify political neutrality by analyzing abstractive text summarization of news articles on polarizing issues. Their core contribution is the introduction of a "Polarization Index (P)." This index is calculated by measuring the difference in classifier effectiveness between distinguishing between neutral and politically aligned summaries. Specifically, it quantifies how well a classifier can differentiate between Democrat-leaning and neutral summaries, and then subtracts the difference between Republican-leaning and neutral summaries. This metric aims to capture the degree to which default summaries diverge from a truly neutral stance.

In addition to the Polarization Index, the researchers also employed a diverging vocabulary analysis. This method identifies and compares the frequency of specific lexical choices in summaries aligned with Democrat and Republican viewpoints, providing qualitative insights into the language used by LLMs.

Datasets

The study utilized a substantial corpus of 20,344 news articles spanning from 2019 to 2024. These articles covered five key pressing issues in US politics: abortion, gun control/rights, healthcare, immigration, and LGBTQ+ rights. The dataset was curated from various news outlets to ensure a balanced representation of political viewpoints. No new benchmark datasets were introduced; rather, an existing corpus was leveraged for analysis.

Key Insights and Ideas

  • Subtle Biases in "Neutral" Summaries: Even when explicitly prompted for neutral summaries, LLMs demonstrate a consistent tendency to lean towards liberal viewpoints. This suggests that "neutrality" in LLM outputs is not always objective and can be influenced by underlying training data or model architectures.
  • LLM Monoculture: The study found a significant convergence in the vocabulary used by different LLMs across various political topics. This "LLM monoculture" implies that these models may be homogenizing the framing of political discourse, potentially limiting the diversity of perspectives presented to users.
  • Implications for Public Opinion: The consistent bias and vocabulary convergence raise concerns about LLMs' potential to warp political perceptions, especially among younger generations who increasingly rely on these models for information. This could lead to the formation of echo chambers and the reinforcement of existing, potentially biased, political narratives.

Key Results

The Polarization Index (P) consistently showed a bias towards Democrat-aligned perspectives across all topics and models tested. The mean P values were negative for all topics, indicating this trend.

  • Gun Control/Rights exhibited the highest mean polarization (-3.63%) and maximum polarization (-9.49%).
  • Healthcare also showed significant polarization, with a mean P of -3.09% and a maximum of -6.14%.
  • Abortion had a mean P of -2.79% and a maximum of -4.30%.
  • Immigration had a mean P of -2.57% and a maximum of -5.66%.
  • LGBTQ+ rights showed the least polarization among the tested topics, with a mean P of -2.05% and a maximum of -2.98%.

Furthermore, the study found a high overlap in vocabulary: 55% for Democrat-leaning representations and 52% for Republican-leaning representations, reinforcing the idea of LLM monoculture.

Limitations and Future Work

Limitations:

  • The study focuses on abstractive summarization, and the findings might not directly generalize to other LLM tasks.
  • The analysis is limited to five specific US political issues.

Future Work:

The authors call for:

  1. Increased scrutiny of AI-generated content in political discourse.
  2. Development of more robust techniques to ensure political neutrality in LLMs.
  3. Greater transparency from LLM developers and education of users about existing biases.
  4. Further research into methods for detecting and mitigating political bias in LLMs.

Summary updated

BibTeX
@inproceedings{vijay2025when,
  author = {Supriti Vijay and Aman Priyanshu and Ashiqur R. KhudaBukhsh},
  title = {{When Neutral Summaries Are Not That Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35308},
}
Figure