The Cultural Psychology of Large Language Models: Is ChatGPT a Holistic or Analytic Thinker?

Chuanyang Jin, Songyang Zhang, Tianmin Shu, Zhihan CuiarXiv 2023 · OpenReview

Abstract

The prevalent use of Large Language Models (LLMs) has necessitated studying their mental models, yielding noteworthy theoretical and practical implications. Current research has demonstrated that state-of-the-art LLMs, such as ChatGPT, exhibit certain theory of mind capabilities and possess relatively stable personality traits. However, cognitive process features, which differ significantly between Eastern (holistic) and Western (analytic) cultures, remain under-explored. In this research, we probed the cultural cognitive traits of ChatGPT using the Analysis-Holism Scale (AHS), Triadic Categorization Task (TCT), Dialectical Self Scale (DSS), and Self-construal Scale (SCS). Our findings indicate that while ChatGPT consistently tends towards Eastern holistic thinking in cognitive processes, it does not significantly lean towards the East or West in value judgments, often exhibiting a Western-leaning independent self-construal. We suggest these results stem from both the training paradigm and the training data in LLM development.

Introduction

As Large Language Models (LLMs) like ChatGPT become deeply integrated into societal functions, understanding their underlying "mental models" is critical. A key dimension of human psychology is the cultural divide in cognitive styles: Analytic thinking, common in Western cultures, focuses on discrete objects and formal logic; Holistic thinking, prevalent in Eastern cultures, emphasizes context, relationships, and the acceptance of contradiction (naive dialecticism). This study investigates where ChatGPT falls on this spectrum and whether its cognitive style is consistent across different psychological assessments.

Methods

We evaluated OpenAI’s GPT-3.5 and GPT-4 using four established psychological scales:

  1. Cognitive Process Measures: The Analysis-Holism Scale (AHS) and the Triadic Categorization Task (TCT), an implicit task where models group objects based on either category (analytic) or relationship (holistic).
  2. Value Judgment Measures: The Dialectical Self Scale (DSS) and the Self-Construal Scale (SCS), which measures independent vs. interdependent views of the self.

To ensure robustness, models were tested across multiple "temperatures" (randomness settings) and various prompt paraphrases. Results were compared against established human norms from psychological literature.

Results

Our analysis revealed a distinct "split personality" in ChatGPT's cultural profile.

Cognitive Style: A Holistic Bias

In cognitive process tests, both models showed a significant inclination toward Eastern holistic thinking. On the AHS, GPT-3.5 (M=5.35) and GPT-4 (M=5.01) scored significantly higher than the human norm (M=4.87, $p < 0.001$). In the TCT grouping task, the models chose relational (holistic) pairings 85-87% of the time, exceeding the mean of most Chinese provinces in comparison data.

Value Judgments: A Western Lean

Conversely, in value-based assessments, the models often leaned Western. On the SCS-Independent scale, GPT-3.5 scored a near-perfect 6.98/7.0, significantly higher than the human norm (4.73). While GPT-4 (5.32) was closer to human norms, it still leaned heavily toward Western individualism.

Scores across measurement scales under varying temperatures and prompts for GPT-3.5 and GPT-4.

Robustness and Model Differences

The cultural profiles remained consistent across varying temperatures and prompt paraphrases, suggesting these are stable traits rather than stochastic "hallucinations." Notably, GPT-4 generally aligned more closely with human norms than GPT-3.5, likely due to more rigorous Reinforcement Learning from Human Feedback (RLHF) and alignment tuning.

Discussion

The holistic nature of ChatGPT's cognitive processing may be an inherent byproduct of the Transformer architecture, which predicts tokens based on the entire preceding context—a fundamentally holistic approach. However, the disconnect between process and value suggests that training data and RLHF act as cultural filters, potentially imposing Western values on a holistically-processing engine.

Future research should expand this inquiry to LLMs developed in different cultural contexts (e.g., Baidu’s Ernie Bot) to determine if holistic processing is universal to the Transformer architecture or specific to OpenAI’s development pipeline. Understanding these biases is essential for the responsible deployment of AI in diverse global markets.

Summary updated

Abstract

The prevalent use of Large Language Models (LLMs) has necessitated studying their mental models, yielding noteworthy theoretical and practical implications. Current research has demonstrated that state-of-the-art LLMs, such as ChatGPT, exhibit certain theory of mind capabilities and possess relatively stable Big Five and/or MBTI personality traits. In addition, cognitive process features form an essential component of these mental models. Research in cultural psychology indicated significant differences in the cognitive processes of Eastern and Western people when processing information and making judgments. While Westerners predominantly exhibit analytical thinking that isolates things from their environment to analyze their nature independently, Easterners often showcase holistic thinking, emphasizing relationships and adopting a global viewpoint. In our research, we probed the cultural cognitive traits of ChatGPT. We employed two scales that directly measure the cognitive process: the Analysis-Holism Scale (AHS) and the Triadic Categorization Task (TCT). Additionally, we used two scales that investigate the value differences shaped by cultural thinking: the Dialectical Self Scale (DSS) and the Self-construal Scale (SCS). In cognitive process tests (AHS/TCT), ChatGPT consistently tends towards Eastern holistic thinking, but regarding value judgments (DSS/SCS), ChatGPT does not significantly lean towards the East or the West. We suggest that the result could be attributed to both the training paradigm and the training data in LLM development. We discuss the potential value of this finding for AI research and directions for future research.
BibTeX
@misc{jin2023cultural,
  author = {Chuanyang Jin and Songyang Zhang and Tianmin Shu and Zhihan Cui},
  title = {{The Cultural Psychology of Large Language Models: Is ChatGPT a Holistic or Analytic Thinker?}},
  year = {2023},
  eprint = {2308.14242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.14242},
}
Figure