The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs

Pengrui Han, Rafal Kocielnik, Peiyang Song, Ramit Debnath, Dean Mobbs, Anima Anandkumar, R. Michael Alvarez

Abstract

Personality traits have long been studied as predictors of human behavior. Recent advances in Large Language Models (LLMs) suggest similar patterns may emerge in artificial systems, with advanced LLMs displaying consistent behavioral tendencies resembling human traits like agreeableness and self-regulation. Understanding these patterns is crucial, yet prior work primarily relied on simplified self-reports and heuristic prompting, with little behavioral validation. In this study, we systematically characterize LLM personality across three dimensions: (1) the dynamic emergence and evolution of trait profiles throughout training stages; (2) the predictive validity of self-reported traits in behavioral tasks; and (3) the impact of targeted interventions, such as persona injection, on both self-reports and behavior. Our findings reveal that instructional alignment (e.g., RLHF, instruction tuning) significantly stabilizes trait expression and strengthens trait correlations in ways that mirror human data. However, these self-reported traits do not reliably predict behavior, and observed associations often diverge from human patterns. While persona injection successfully steers self-reports in the intended direction, it exerts little or inconsistent effect on actual behavior. By distinguishing surface-level trait expression from behavioral consistency, our findings challenge assumptions about LLM personality and underscore the need for deeper evaluation in alignment and interpretability.

Introduction

Researchers and users often anthropomorphize Large Language Models (LLMs), using standard human psychological tests like the Big Five Inventory to "profile" them. However, it remains unclear if an LLM that claims to be "agreeable" or "conscientious" actually behaves that way when faced with functional scenarios. This study investigates the "Personality Illusion": the gap between surface-level linguistic self-expression and deep behavioral consistency.

Experimental Framework

The researchers systematically characterized LLM personality across three research questions (RQs) using a pipeline that tracks models from pre-training through post-training alignment and persona injection.

The experimental pipeline evaluates models across three dimensions: trait origin, behavioral manifestation, and controllability via persona injection.

The study evaluated 18 models, including the LLaMA 3 family, Qwen 2.5, Mistral, Claude 3.7 Sonnet, and GPT-4o. Behavioral probes included the Columbia Card Task (Risk-Taking), the Implicit Association Test (Social Bias), and an Asch-style conformity paradigm (Sycophancy).

Key Insights: Alignment as a "Crystallizer"

Instructional alignment (RLHF/SFT) significantly stabilizes a model's reported persona. Variability in trait expression dropped by approximately 40% after alignment, making models appear more "human-like" and consistent in their self-descriptions. Aligned models consistently reported higher Openness, Agreeableness, and Self-Regulation, and lower Neuroticism compared to their base versions.

The Great Dissociation

Despite the stability of self-reports, there is a fundamental disconnect between what a model says about itself and what it does. The study found that only about 24% of trait-task associations were statistically significant. Furthermore, of those significant associations, only 52% aligned with human psychological patterns—essentially the same as random chance.

Alignment between self-reported traits and behavioral tasks across different models and tasks, highlighting the significant gap to 100% alignment.

As shown in the figure above, most models fail to reach meaningful alignment between their reported traits and behavioral tasks. Only the largest "frontier" models, such as Qwen-235B, showed modest, statistically significant alignment, suggesting that behavioral consistency might be an emergent property of scale, though it remains weak overall.

Persona Injection is Skin-Deep

When told to adopt a specific persona (e.g., "be agreeable"), models successfully shifted their self-report scores. However, this had minimal to no effect on their actual behavior. For example, injecting an "Agreeable" persona increased self-reported agreeableness scores ($\beta = 3.95, p < .001$) but failed to significantly modulate sycophancy ($\beta = 0.03, p = 0.67$). This suggests LLMs simulate personality through surface-level pattern matching rather than internalized motivations.

Conclusion

This paper serves as a cautionary tale against taking an LLM's "personality" at face value. The authors argue for "Behaviorally-Grounded Alignment," suggesting that instead of just training models to sound better, we should use Reinforcement Learning from Behavioral Feedback (RLBF) to reward consistent performance in psychologically grounded tasks. By distinguishing between linguistic plausibility and functional reliability, we can move toward more robust and interpretable AI systems.

Summary updated

Abstract

Personality traits have long been studied as predictors of human behavior. Recent advances in Large Language Models (LLMs) suggest similar patterns may emerge in artificial systems, with advanced LLMs displaying consistent behavioral tendencies resembling human traits like agreeableness and self-regulation. Understanding these patterns is crucial, yet prior work primarily relied on simplified self-reports and heuristic prompting, with little behavioral validation. In this study, we systematically characterize LLM personality across three dimensions: (1) the dynamic emergence and evolution of trait profiles throughout training stages; (2) the predictive validity of self-reported traits in behavioral tasks; and (3) the impact of targeted interventions, such as persona injection, on both self-reports and behavior. Our findings reveal that instructional alignment (e.g., RLHF, instruction tuning) significantly stabilizes trait expression and strengthens trait correlations in ways that mirror human data. However, these self-reported traits do not reliably predict behavior, and observed associations often diverge from human patterns. While persona injection successfully steers self-reports in the intended direction, it exerts little or inconsistent effect on actual behavior. By distinguishing surface-level trait expression from behavioral consistency, our findings challenge assumptions about LLM personality and underscore the need for deeper evaluation in alignment and interpretability.
BibTeX
@inproceedings{han2026personality,
  author = {Pengrui Han and Rafal Kocielnik and Peiyang Song and Ramit Debnath and Dean Mobbs and Anima Anandkumar and R. Michael Alvarez},
  title = {{The Personality Illusion: Revealing Dissociation Between Self-Reports \& Behavior in LLMs}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=6OMZEKarO7},
}

Loading related content...

Figure