Abstract
Psychology research has long explored aspects of human personality like extroversion, agreeableness and emotional stability, three of the personality traits that make up the ‘Big Five’. Categorizations like the ‘Big Five’ are commonly used to assess and diagnose personality types. In this work, we explore whether text generated from large language models exhibits consistency in it’s perceived ‘Big Five’ personality traits. For example, is a language model such as GPT2 likely to respond in a consistent way if asked to go out to a party? We also show that when exposed to different types of contexts (such as personality descriptions, or answers to diagnostic questions about personality traits), language models such as BERT and GPT2 consistently identify and mirror personality markers in those contexts. This behavior illustrates an ability to be manipulated in a predictable way (with correlations up to 0.84 between intended and realized changes in personality traits), and frames them as tools for controlling personas in applications such as dialog systems. We contribute two data-sets of personality descriptions of humans subjects.
Introduction
As Large Language Models (LLMs) like BERT and GPT-2 become increasingly integrated into human-AI interaction, users naturally tend to anthropomorphize them, assigning them distinct "personalities." However, these perceived traits are often accidental byproducts of training data biases rather than intentional design. This research addresses whether LLMs exhibit consistent, measurable personality traits and, more importantly, whether these traits can be deliberately steered using simple textual context.
By framing personality through the "Big Five" (OCEAN) psychometric lens—Extroversion, Agreeableness, Conscientiousness, Emotional Stability, and Openness to Experience—the study demonstrates that models consistently mirror personality markers provided in prompts. This ability to manipulate personas predictably has significant implications for creating safer, more effective AI agents, such as supportive clinical assistants or professional email tools.
Methodology
To measure personality traits in AI, the researchers adapted the IPIP 50-item personality assessment, a gold standard in human psychology. Since models cannot "take tests" in the traditional sense, the questionnaire was converted into a fill-in-the-blank format (e.g., "I am {blank} the life of the party").
- BERT was evaluated based on the probability of the masked token.
- GPT-2 was evaluated by comparing the likelihood of sentences using different adverb choices (e.g., always, often, rarely, never).
The study introduced three types of context to manipulate the models: assessment item context (using test questions as prefixes), Reddit context (scraped human personality descriptions), and psychometric survey data (self-reported descriptions from human subjects).
Data Sources
The study utilized several key datasets to establish benchmarks and evaluate model performance:
- IPIP (International Personality Item Pool): Source for the 50-item diagnostic questions.
- Open-Psychometrics: A dataset of 1,015,000 individuals used to establish human population norms.
- Reddit Dataset: 1,119 first-person responses from subreddits like r/AskReddit describing personal traits.
- MTurk Survey Dataset: 404 responses from Amazon Mechanical Turk where subjects completed the Big Five test and wrote personality descriptions.
Key Results and Insights
The researchers found that LLMs are not "blank slates"; they possess inherent "base" personalities. For instance, GPT-2 scored higher than the human median in Conscientiousness but lower in Agreeableness. However, these base traits are highly malleable through the "Mirroring Effect."
The manipulation proved remarkably predictable. The study found median Pearson correlations of 0.84 for BERT and 0.81 for GPT-2 between the expected change and the observed change in personality traits. For traits like Conscientiousness and Emotional Stability, models were able to span the entire human range (0 to 100th percentile) depending on the context provided. Furthermore, these shifts were reflected in free-form text generation; an "Agreeable" prompt led GPT-2 to generate text emphasizing positive social connections.
Limitations and Future Work
While the results are promising, several challenges remain. The study focused on smaller models (BERT-base and GPT-2 124M), and it is unclear how these findings scale to massive models like GPT-4. Additionally, models occasionally struggled with linguistic nuances like double negatives, leading to unpredictable scores. Future research will explore more complex personality taxonomies and investigate how specific pre-training data shapes inherent model biases.
Summary updated