Abstract
The emergence of large-scale pretrained language models has revolutionized the capabilities of new AI application, especially in the realm of crafting chatbots with distinct personas. Given the "stimulus-response" nature of chatbots, this paper unveils an innovative open-ended interview-style approach for personality assessment on role-playing chatbots, which offers a richer comprehension of their intrinsic personalities. We conduct personality assessments on 32 role-playing chatbots created by the ChatHaruhi library, across both the Big Five and MBTI dimensions, and measure their alignment with human perception. Evaluation results underscore that modern role-playing chatbots based on LLMs can effectively portray personality traits of corresponding characters, with an alignment rate of 82.8% compared with human-perceived personalities. Besides, we also suggest potential strategies for shaping chatbots' personalities. Hence, this paper serves as a cornerstone study for role-playing chatbots that intersects computational linguistics and psychology.
Introduction
As Large Language Models (LLMs) power increasingly sophisticated role-playing agents (RPAs), evaluating their "character fidelity" has become a major hurdle. Most existing benchmarks focus on knowledge or linguistic style, often ignoring the underlying personality—the consistent patterns of thought and behavior that define a character.
Previous attempts to measure personality usually relied on "self-report" scales (like the Big Five or MBTI), where the agent is asked to rate itself on a Likert scale. This approach often fails because asking a character to "take a personality test" breaks role-play immersion, and LLMs may select options based on training biases rather than their latent personality expressed through dialogue.
Methodology: The INCHARACTER Framework
The authors propose INCHARACTER, a two-stage framework inspired by clinical psychological interviews. Instead of a survey, the framework transforms scale items into open-ended questions. For example, the Big Five item "Values artistic, aesthetic experiences" becomes "Do you value artistic, aesthetic experiences?" The RPA responds in character, drawing from its persona data and memories.
In the assessment stage, a separate "Interviewer LLM" (typically GPT-4) analyzes the interview transcripts using two scoring methods:
- Option Conversion (OC): The judge converts a single response into a Likert value.
- Expert Rating (ER): The judge reviews a batch of responses for a specific personality dimension and assigns a holistic score.
To further improve accuracy, the authors introduced Character-specific Question Adaptation, minimally tweaking questions to fit a character's world (e.g., changing "movies" to "tales and legends" for a medieval fantasy character) to avoid "out-of-character" hallucinations.
Experimental Setup
The study utilized a comprehensive suite of 32 distinct personas from popular fiction (e.g., Harry Potter, Genshin Impact) and real-world celebrities. These were evaluated against 14 widely used psychological scales, including the Big Five Inventory (BFI) and 16Personalities. Ground truth personality labels were sourced from the Personality Database (PDb) and expert human annotators.
Key Results and Insights
- High Accuracy: Using GPT-4 as an interviewer, the framework achieved an alignment rate of up to 82.8% (with specific configurations reaching 80.7% accuracy in general benchmarks) compared to human benchmarks.
- Interviews > Surveys: The interview-based approach yielded significantly higher consistency and better alignment with human labels than self-report methods. It allows the agent's personality to emerge naturally through narrative.
- The "Pleaser" Problem: Some commercial models showed lower personality fidelity because they were tuned to be "compliant and pleasing" to the user, an "AI assistant" bias that can override the character's intended personality.
- Data Importance: Combining character descriptions with memories (past dialogues) is essential for high fidelity; descriptions alone are often insufficient to capture the "soul" of the character.
Discussion and Limitations
While the framework shows promise, the authors note several limitations. The study uses static personality labels, whereas characters often evolve over time. Additionally, the evaluation relies heavily on the "Interviewer LLM," which may have its own biases. Ethical concerns regarding testing for "Dark Triad" traits also require careful safeguards to prevent the generation of harmful content. Future work will focus on scaling the benchmark to a wider variety of personas and exploring personality dynamics.
Summary updated