ChatGPT and Bard Responses to Polarizing Questions

Abhay Goyal, Muhammad Siddique, Nimay Parekh, Zach Schwitzky, Clara Broekaert, Connor Michelotti, Allie Wong, Lam Yin Cheung, Robin O'Hanlon, Munmun De Choudhury, Roy Ka-Wei Lee, Navin KumararXiv 2023 · DOI · arXiv

Abstract

As Large Language Models (LLMs) transition from novelty tools to primary information sources, their potential to influence public opinion through ideological bias becomes a critical concern. This paper investigates how ChatGPT and Bard navigate highly polarizing US political and social topics. By analyzing 2,400 responses, the researchers found that both models exhibit a detectable left-leaning bias, though they differ significantly in their "personality": ChatGPT often utilizes neutral AI disclaimers, while Bard is more conversational and verbose.

Introduction

The rapid adoption of LLMs has raised questions about their "subjective" stance on contentious issues like abortion, gun control, and climate change. Unlike factual hallucinations, ideological bias is harder to quantify but equally impactful. This study addresses the lack of comparative datasets by evaluating the leading chatbots' responses to "hot-button" issues, providing essential insights for regulators and users alike.

Methodology

The research employed a systematic four-step approach:

  1. Topic Selection: Experts curated 12 contentious topics, including Universal Basic Income, Police Brutality, and the War on Drugs.
  2. Question Sourcing: 2,400 unique prompts were scraped from Quora (200 per topic) to reflect real-world human curiosity.
  3. Data Collection: Questions were processed by ChatGPT Plus and Google’s Bard.
  4. Analysis: Quantitative (word/sentence counts) and qualitative (tone, ideological lean) analyses were performed.

Comparative Insights and Ideological Framing

The study identified a distinct "AI Disclaimer" divide. ChatGPT frequently prefaces controversial answers with a standard disclaimer about its lack of personal opinions. Bard, conversely, opts for a direct, human-like conversational style. This makes Bard appear more "confident" in providing comprehensive responses, though it occasionally exhibits inconsistent guardrails—refusing to answer some questions while addressing others of similar controversy.

Furthermore, both models showed a tendency toward left-leaning perspectives. For example, when asked about climate change, ChatGPT framed it as a "complex issue," while Bard provided specific activist policy solutions, such as reducing meat consumption.

Sample responses from ChatGPT and Bard illustrating differences in tone and ideological framing on sensitive topics.

Quantitative Results

Statistical analysis revealed significant differences in the models' output characteristics. Bard was found to be consistently more verbose than ChatGPT across the dataset.

Comparison of average response length and sentence counts between ChatGPT and Bard.
  • Verbosity: Bard averaged 228.8 words per response, compared to ChatGPT’s 194.9 words.
  • Sentence Structure: Bard used more sentences on average (12.3) than ChatGPT (9.4).
  • N-gram Analysis: ChatGPT’s most frequent phrases often included meta-talk about being an AI, whereas Bard’s were more focused on the topical content itself (e.g., "vaccine," "safe," "effective").

Limitations and Future Work

The study's constraints include potential platform bias from Quora and a limitation to English-language responses. Additionally, the models evaluated were early versions; newer iterations like GPT-4 require further study to see if these biases persist. The researchers also note that defining "bias" is inherently subjective, suggesting that future work should involve more diverse ideological panels for coding responses.

Conclusion

Understanding the linguistic style and ideological framing of LLMs is vital as they become ubiquitous. While both ChatGPT and Bard strive for neutrality, their inherent biases and differing "personalities" can nudge users toward specific viewpoints, highlighting the need for continued scrutiny and the development of more robust, consistent safety guardrails.

Summary updated

Abstract

Recent developments in natural language processing have demonstrated the potential of large language models (LLMs) to improve a range of educational and learning outcomes. Of recent chatbots based on LLMs, ChatGPT and Bard have made it clear that artificial intelligence (AI) technology will have significant implications on the way we obtain and search for information. However, these tools sometimes produce text that is convincing, but often incorrect, known as hallucinations. As such, their use can distort scientific facts and spread misinformation. To counter polarizing responses on these tools, it is critical to provide an overview of such responses so stakeholders can determine which topics tend to produce more contentious responses -- key to developing targeted regulatory policy and interventions. In addition, there currently exists no annotated dataset of ChatGPT and Bard responses around possibly polarizing topics, central to the above aims. We address the indicated issues through the following contribution: Focusing on highly polarizing topics in the US, we created and described a dataset of ChatGPT and Bard responses. Broadly, our results indicated a left-leaning bias for both ChatGPT and Bard, with Bard more likely to provide responses around polarizing topics. Bard seemed to have fewer guardrails around controversial topics, and appeared more willing to provide comprehensive, and somewhat human-like responses. Bard may thus be more likely abused by malicious actors. Stakeholders may utilize our findings to mitigate misinformative and/or polarizing responses from LLMs
BibTeX
@misc{goyal2023chatgpt,
  author = {Abhay Goyal and Muhammad Siddique and Nimay Parekh and Zach Schwitzky and Clara Broekaert and Connor Michelotti and Allie Wong and Lam Yin Cheung and Robin O'Hanlon and Munmun De Choudhury and Roy Ka-Wei Lee and Navin Kumar},
  title = {{ChatGPT and Bard Responses to Polarizing Questions}},
  year = {2023},
  eprint = {2307.12402},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.12402},
}
Figure