Personality Matters: User Traits Predict LLM Preferences in Multi-Turn Collaborative Tasks

Sarfaroz Yunusov, Kaige Chen, Kazi Nishat Anwar, Ali EmamiEMNLP 2025

TL;DR

This paper reveals that users' personality types significantly influence their preferences for different Large Language Models (LLMs) in collaborative tasks. Rationals prefer GPT-4 for goal-oriented tasks, while Idealists favor Claude 3.5 for creative and analytical work, highlighting how personality-driven analysis offers a richer understanding of LLM performance than traditional aggregate metrics.

What Problem Does This Paper Address?

As LLMs become increasingly integrated into our daily workflows, they are often used in multi-turn collaborative settings where users refine prompts and guide the AI. However, most LLM evaluations focus on single-turn interactions and fail to account for individual user differences. This paper addresses the critical question of whether users with distinct personality traits exhibit systematic preferences for different LLMs during these extended collaborative tasks, and how these preferences impact their perception of helpfulness. Understanding these user-specific factors is crucial for developing more effective and personalized LLM interactions.

Methods

The researchers conducted a study involving 32 participants, evenly distributed across four Keirsey personality types: Rationals, Idealists, Guardians, and Artisans. Participants engaged in four distinct collaborative tasks (data analysis, creative writing, information retrieval, and writing assistance) using either GPT-4 or Claude 3.5. To ensure participants were blind to the model they were using, a custom web application with an identical interface was employed. After each task, participants rated the LLM's helpfulness on a 1-5 scale. Sentiment analysis of qualitative feedback was also performed using a BERT-based model and manual review to corroborate quantitative findings.

Datasets

The study utilized a synthetic dataset for the "Data Analysis – Student Loan Dataset" task. This dataset contained 500 student loan records with attributes such as age, gender, loan purpose, and account balances. No new benchmarks were introduced; the focus was on evaluating existing LLMs with real-world collaborative tasks.

Key Insights and Ideas

  • Personality is a significant predictor of LLM preference: The study moves beyond generic LLM evaluations to demonstrate that individual user traits are not just a minor factor but a primary driver of how helpful users perceive an LLM to be.
  • Aggregate metrics can be misleading: The finding that overall helpfulness ratings for GPT-4 and Claude 3.5 were nearly identical highlights how traditional evaluation methods can mask crucial, personality-driven differences in user experience.
  • Task type interacts with personality: Preferences are not absolute but are influenced by the nature of the task. Rationals excel with GPT-4 in goal-oriented tasks, while Idealists find Claude 3.5 more beneficial for creative and analytical endeavors.
  • Qualitative feedback provides deeper context: Sentiment analysis and manual review of user feedback confirmed the quantitative findings, revealing that different personality types use distinct criteria (e.g., precision vs. holistic understanding) to assess LLM helpfulness.
  • Prompting strategy matters: Concise prompts led to better outcomes in data analysis tasks, suggesting that how users interact with LLMs can significantly impact success, especially when combined with personality-driven model preferences.

Key Results

  • Rationals strongly preferred GPT-4 (4.3/5) over Claude 3.5 (3.7/5), particularly for data analysis.
  • Idealists showed the opposite preference, favoring Claude 3.5 (4.4/5) over GPT-4 (3.6/5), especially for creative and analytical tasks.
  • Guardians and Artisans exhibited task-dependent preferences, with Guardians favoring GPT-4 for data analysis but Claude 3.5 for information retrieval.
  • Sentiment analysis aligned with preference data, showing more positive sentiment towards GPT-4 from Rationals and towards Claude 3.5 from Idealists and Guardians.
  • Overall helpfulness ratings were nearly identical for GPT-4 (3.87/5) and Claude 3.5 (3.85/5) when aggregated across all participants.

Limitations and Future Work

The authors acknowledge several limitations:

  • Demographic Scope: Participants were primarily university students, limiting the generalizability of findings to broader populations. Future work should expand to more diverse demographics.
  • Subjectivity in Ratings: Helpfulness is inherently subjective. The study mitigated this by using multiple measures (ratings, sentiment, qualitative feedback) but acknowledges it as a fundamental challenge in LLM evaluation.
  • Model Selection: The study focused on two high-performing LLMs (GPT-4 and Claude 3.5) due to experimental feasibility. Future research could explore how personality traits interact with a wider range of LLM capabilities.
  • Personality Classification: The Keirsey Temperament Sorter provides categorical clarity but may lack the psychometric precision of dimensional models. A dimensional approach would require a much larger sample size.

Future work should investigate whether these personality-driven patterns hold across broader populations and explore how LLMs could be adapted to user personalities to enhance collaborative outcomes.

Summary updated

BibTeX
@inproceedings{yunusov2025personality,
  author = {Sarfaroz Yunusov and Kaige Chen and Kazi Nishat Anwar and Ali Emami},
  title = {{Personality Matters: User Traits Predict LLM Preferences in Multi-Turn Collaborative Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure