Abstract
As Large Language Models (LLMs) transition from novelty tools to primary information sources, their potential to influence public opinion through ideological bias becomes a critical concern. This paper investigates how ChatGPT and Bard navigate highly polarizing US political and social topics. By analyzing 2,400 responses, the researchers found that both models exhibit a detectable left-leaning bias, though they differ significantly in their "personality": ChatGPT often utilizes neutral AI disclaimers, while Bard is more conversational and verbose.
Introduction
The rapid adoption of LLMs has raised questions about their "subjective" stance on contentious issues like abortion, gun control, and climate change. Unlike factual hallucinations, ideological bias is harder to quantify but equally impactful. This study addresses the lack of comparative datasets by evaluating the leading chatbots' responses to "hot-button" issues, providing essential insights for regulators and users alike.
Methodology
The research employed a systematic four-step approach:
- Topic Selection: Experts curated 12 contentious topics, including Universal Basic Income, Police Brutality, and the War on Drugs.
- Question Sourcing: 2,400 unique prompts were scraped from Quora (200 per topic) to reflect real-world human curiosity.
- Data Collection: Questions were processed by ChatGPT Plus and Google’s Bard.
- Analysis: Quantitative (word/sentence counts) and qualitative (tone, ideological lean) analyses were performed.
Comparative Insights and Ideological Framing
The study identified a distinct "AI Disclaimer" divide. ChatGPT frequently prefaces controversial answers with a standard disclaimer about its lack of personal opinions. Bard, conversely, opts for a direct, human-like conversational style. This makes Bard appear more "confident" in providing comprehensive responses, though it occasionally exhibits inconsistent guardrails—refusing to answer some questions while addressing others of similar controversy.
Furthermore, both models showed a tendency toward left-leaning perspectives. For example, when asked about climate change, ChatGPT framed it as a "complex issue," while Bard provided specific activist policy solutions, such as reducing meat consumption.
Quantitative Results
Statistical analysis revealed significant differences in the models' output characteristics. Bard was found to be consistently more verbose than ChatGPT across the dataset.
- Verbosity: Bard averaged 228.8 words per response, compared to ChatGPT’s 194.9 words.
- Sentence Structure: Bard used more sentences on average (12.3) than ChatGPT (9.4).
- N-gram Analysis: ChatGPT’s most frequent phrases often included meta-talk about being an AI, whereas Bard’s were more focused on the topical content itself (e.g., "vaccine," "safe," "effective").
Limitations and Future Work
The study's constraints include potential platform bias from Quora and a limitation to English-language responses. Additionally, the models evaluated were early versions; newer iterations like GPT-4 require further study to see if these biases persist. The researchers also note that defining "bias" is inherently subjective, suggesting that future work should involve more diverse ideological panels for coding responses.
Conclusion
Understanding the linguistic style and ideological framing of LLMs is vital as they become ubiquitous. While both ChatGPT and Bard strive for neutrality, their inherent biases and differing "personalities" can nudge users toward specific viewpoints, highlighting the need for continued scrutiny and the development of more robust, consistent safety guardrails.
Summary updated