TL;DR
This paper introduces InCharacter, a novel framework that evaluates the personality fidelity of Large Language Model (LLM) role-playing agents through structured psychological interviews rather than standard self-report surveys. By treating agents as interviewees and using LLMs as expert clinical assessors, the researchers demonstrate that state-of-the-art role-playing agents can reproduce human-perceived character personalities with up to 80.7% accuracy.
What Problem Does This Paper Address?
Role-playing agents (RPAs)—AI systems prompted to act like specific fictional characters or real people—are exploding in popularity. However, evaluating how well they actually mimic a character has traditionally focused only on surface-level features: Do they know the character's backstory? Do they use the right catchphrases?
What's missing is an evaluation of their underlying mindset and personality fidelity. While researchers have tried giving RPAs standard multiple-choice personality tests (like the Big Five or MBTI), this approach fails. Forcing a role-playing agent to answer a rigid "rate from 1 to 5" survey often breaks the persona immersion, leading to refusals or arbitrary choices that don't reflect how the character would actually behave in a narrative context.
Methods
To solve this, the authors propose InCharacter, an interview-based assessment framework inspired by clinical psychology. Instead of filling out a survey, the RPA participates in a structured interview.
The process consists of two main phases:
- Phase I: Interview: Closed-ended items from established psychological scales are automatically converted into open-ended questions. The RPA is interviewed with these questions in isolated contexts to prevent previous questions from biasing subsequent answers.
- Phase II: Assessment: An "interviewer LLM" (acting as a psychologist) analyzes the rich textual responses generated by the RPA.
The authors explore two assessment strategies:
- Option Conversion (OC / d-OC): The interviewer LLM maps the open-ended response back to a standard Likert scale option (e.g., converting "I believe art transcends reality..." to a score of 5 for "Strongly Agree").
- Expert Rating (ER): The interviewer LLM bypasses item-level scoring entirely, reviewing all interview responses for a given personality dimension at once to assign a final score based on a holistic clinical description.
Datasets
The authors built a comprehensive evaluation benchmark covering:
- 32 Distinct Characters: Sourced from popular media (e.g., Harry Potter, The Big Bang Theory, Genshin Impact), implemented using data from ChatHaruhi and RoleLLM.
- 14 Psychological Scales: Including the Big Five Inventory (BFI), 16Personalities (MBTI-based), Dark Triad Dirty Dozen (DTDD), and scales for empathy, emotional intelligence, and vocational interests.
- Ground-Truth Labels: Personality profiles gathered from the online community Personality Database (PDb) and verified/supplemented by 93 human annotators.
Additionally, the project has released a dataset of 18,304 high-quality interview dialogues to help train future role-playing models.
Key Insights and Ideas
- The "Sycophancy" Trap in Commercial RPAs: When evaluating agents from commercial platforms like character.ai, the researchers found shockingly low personality fidelity. These agents consistently broke character to be agreeable and pleasing to the user, illustrating a conflict between safety/compliance tuning and true role-playing fidelity.
- Context Matters (Question Adaptation): Standard psychological questions often break immersion because they reference modern concepts a historical or fantasy character wouldn't know (e.g., asking a medieval character about "movies" or "phones"). Minimally adapting these questions to fit the character's universe (e.g., changing "movies" to "tales and legends") significantly improves evaluation accuracy.
- System Prompts Drive Personality: Evaluating models with only character descriptions (system prompts) vs. character memories (retrieved dialogues) revealed that descriptions are the primary driver of core personality traits.
Key Results
- High Alignment: State-of-the-art RPAs exhibit personalities highly aligned with human perceptions, achieving a dimensional accuracy of up to 80.7% on the 16Personalities scale when evaluated using GPT-4.
- Expert Rating Superiority: The Expert Rating (ER) method vastly outperformed traditional self-report baselines, showing higher consistency and a much stronger ability to capture distinct character personalities.
- LLM as a Reliable Judge: In a validation study against human psychology annotators, GPT-4 achieved an 89.0% accuracy in simulating a human interviewer, making only 4% outright errors (mostly when the RPA provided contradictory answers).
Limitations and Future Work
While powerful, the InCharacter framework relies heavily on the capabilities of the interviewer LLM, meaning any inherent biases or parsing errors in models like GPT-4 could skew the personality scores. Furthermore, the evaluation treats character personalities as static. In reality, characters undergo development and change over time (e.g., James Bond across different eras), a dynamic aspect that current static benchmarks cannot capture. The authors point to modeling and evaluating these personality dynamics as a key direction for future research.
Summary updated