Large language models and humans converge in judging public figures’ personalities

Xubo Cao, Michał Kosiński

TL;DR

Large language models (LLMs) like ChatGPT-4 can now accurately predict public figures' personalities, mirroring human judgments with surprising fidelity. This capability emerges without explicit training for personality assessment, suggesting LLMs are learning complex social perception patterns from their vast training data. This opens doors for LLMs to serve as efficient proxies for human judgment in social science research.

What Problem Does This Paper Address?

Accurately assessing personality, especially for public figures, has traditionally been a labor-intensive process. It often involves extensive surveys, human raters, and sophisticated statistical models to aggregate and interpret opinions. Previous research has shown that LLMs' internal representations (embeddings) can capture some of these perceived personality traits, but extracting these insights required specialized training and significant effort. The core challenge this paper addresses is whether a general-purpose LLM, without specific fine-tuning for personality assessment, can directly and accurately infer public figures' personalities as perceived by humans.

Methods

The authors leverage ChatGPT-4's ability to follow instructions and generate text. Instead of training a model, they directly prompt ChatGPT-4 with the Ten-Item Personality Inventory (TIPI) questions, adapted to assess public figures. For each public figure, they asked ChatGPT-4 to rate them on a scale of 1 to 7 for agreement with personality statements (e.g., "I see Donald Trump as extraverted, enthusiastic"). To ensure consistency and mitigate potential biases, ChatGPT-4 was reset between each question, and the entire process was repeated 10 times for each figure. The LLM's responses were then aggregated to form personality profiles.

Datasets

The study uses two primary sources of data:

  • Public Figures: 226 public figures were evaluated.
  • Human Ratings: Aggregate personality ratings for these 226 public figures, collected in previous research by Cao and Kosinski (2024) and others, served as the ground truth for evaluating ChatGPT-4's accuracy. These human ratings were based on the Ten-Item Personality Inventory (TIPI).

No new benchmark datasets were introduced; rather, existing human judgment data was used for validation.

Key Insights and Ideas

  • Emergent Personality Perception: LLMs, trained primarily for text prediction, are demonstrating an emergent ability to understand and replicate human perceptions of personality. This suggests that the vast text data they are trained on contains rich information about how people perceive public figures.
  • LLMs as Proxies for Human Judgment: ChatGPT-4's performance rivals, and in some cases surpasses, that of specialized models trained for personality prediction. This indicates LLMs can serve as efficient and cost-effective proxies for human raters in social science research.
  • The "Chinese Room" Analogy: The paper touches upon the philosophical debate of whether LLMs truly "understand" personality or are merely mimicking patterns. The authors suggest that while superficial mimicry is possible, the depth of the LLM's performance might hint at more complex emergent processes.
  • Familiarity Matters: The accuracy of LLM judgments correlates with the public figure's familiarity, likely because well-known individuals have more extensive representations in the LLM's training data, providing more cues for assessment.

Key Results

  • High Correlation with Human Judgments: ChatGPT-4's zero-shot TIPI ratings showed a strong correlation with aggregate human ratings, ranging from r = 0.76 for Openness to r = 0.87 for Agreeableness.
  • Outperforming Previous Models: These correlations were comparable to, and in some cases better than, regression models specifically trained on embeddings to predict human personality ratings (which achieved correlations around r = 0.72 to 0.85).
  • Superior to Individual Human Raters: ChatGPT-4's ratings were better predictors of aggregate human ratings than individual human ratings themselves, which typically correlate at r = 0.56 to 0.66.
  • Consistency: The internal consistency of ChatGPT-4's ratings across multiple runs was very high, with intra-class correlation coefficients ranging from 0.95 to 0.98.
  • Face Validity: The top and bottom-ranked public figures for each trait aligned with common perceptions (e.g., artists and writers perceived as open-minded, autocrats as disagreeable).

Limitations and Future Work

  • Reliance on Training Data: The accuracy of LLM judgments is dependent on the public figure's presence and representation in the LLM's training data. Individuals with limited online presence might not be accurately assessed.
  • "Zero-Shot" vs. Fine-Tuning: While impressive, the study used a zero-shot approach. Further research could explore how fine-tuning LLMs on specific personality datasets might further enhance accuracy.
  • Generalizability: The study focused on public figures. Future work could explore applying this method to assess personalities of individuals not widely known or present in the training data, by analyzing their personal texts.
  • Beyond TIPI: The authors suggest this method could be extended to predict other traits like likability, intelligence, or political orientation.
  • Mitigating Biases: Aggregating judgments from multiple LLMs could help mitigate biases inherent in any single model's training data.

Summary updated

Abstract

ChatGPT-4 and 600 human raters evaluated 226 public figures' personalities using the Ten-Item Personality Inventory. The correlation between ChatGPT-4 and aggregate human ratings ranged from <i>r</i> = 0.76 to 0.87, outperforming the models specifically trained to make such predictions. Notably, the model was not provided with any training data or feedback on its performance. We discuss the potential explanations and practical implications of ChatGPT-4's ability to mimic human responses accurately.
BibTeX
@inproceedings{cao2024large,
  author = {Xubo Cao and Michał Kosiński},
  title = {{Large language models and humans converge in judging public figures’ personalities}},
  booktitle = {PNAS},
  year = {2024},
  doi = {10.1093/pnasnexus/pgae418},
}
Figure