InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews

Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, Yanghua Xiao

TL;DR

This paper introduces InCharacter, a novel framework that evaluates the personality fidelity of Large Language Model (LLM) role-playing agents through structured psychological interviews rather than standard self-report surveys. By treating agents as interviewees and using LLMs as expert clinical assessors, the researchers demonstrate that state-of-the-art role-playing agents can reproduce human-perceived character personalities with up to 80.7% accuracy.

What Problem Does This Paper Address?

Role-playing agents (RPAs)—AI systems prompted to act like specific fictional characters or real people—are exploding in popularity. However, evaluating how well they actually mimic a character has traditionally focused only on surface-level features: Do they know the character's backstory? Do they use the right catchphrases?

What's missing is an evaluation of their underlying mindset and personality fidelity. While researchers have tried giving RPAs standard multiple-choice personality tests (like the Big Five or MBTI), this approach fails. Forcing a role-playing agent to answer a rigid "rate from 1 to 5" survey often breaks the persona immersion, leading to refusals or arbitrary choices that don't reflect how the character would actually behave in a narrative context.

Methods

To solve this, the authors propose InCharacter, an interview-based assessment framework inspired by clinical psychology. Instead of filling out a survey, the RPA participates in a structured interview.

The process consists of two main phases:

  1. Phase I: Interview: Closed-ended items from established psychological scales are automatically converted into open-ended questions. The RPA is interviewed with these questions in isolated contexts to prevent previous questions from biasing subsequent answers.
  2. Phase II: Assessment: An "interviewer LLM" (acting as a psychologist) analyzes the rich textual responses generated by the RPA.

The authors explore two assessment strategies:

  • Option Conversion (OC / d-OC): The interviewer LLM maps the open-ended response back to a standard Likert scale option (e.g., converting "I believe art transcends reality..." to a score of 5 for "Strongly Agree").
  • Expert Rating (ER): The interviewer LLM bypasses item-level scoring entirely, reviewing all interview responses for a given personality dimension at once to assign a final score based on a holistic clinical description.
Figure 2: The framework of INCHARACTER for personality tests on RPAs. Left: Previous methods use self-report scales, which prompt LLMs to select an option directly. Right: INCHARACTER adopts an interview-based approach comprising two phases: the interview and assessment phases. The interview phase elicits the behavioral, cognitive, and emotional patterns of RPAs that reflect their underlying mindsets. The assessment phase measures personalities based on interview results, with two alternative methodologies: option conversion and expert rating.

Datasets

The authors built a comprehensive evaluation benchmark covering:

  • 32 Distinct Characters: Sourced from popular media (e.g., Harry Potter, The Big Bang Theory, Genshin Impact), implemented using data from ChatHaruhi and RoleLLM.
  • 14 Psychological Scales: Including the Big Five Inventory (BFI), 16Personalities (MBTI-based), Dark Triad Dirty Dozen (DTDD), and scales for empathy, emotional intelligence, and vocational interests.
  • Ground-Truth Labels: Personality profiles gathered from the online community Personality Database (PDb) and verified/supplemented by 93 human annotators.

Additionally, the project has released a dataset of 18,304 high-quality interview dialogues to help train future role-playing models.

Key Insights and Ideas

  • The "Sycophancy" Trap in Commercial RPAs: When evaluating agents from commercial platforms like character.ai, the researchers found shockingly low personality fidelity. These agents consistently broke character to be agreeable and pleasing to the user, illustrating a conflict between safety/compliance tuning and true role-playing fidelity.
  • Context Matters (Question Adaptation): Standard psychological questions often break immersion because they reference modern concepts a historical or fantasy character wouldn't know (e.g., asking a medieval character about "movies" or "phones"). Minimally adapting these questions to fit the character's universe (e.g., changing "movies" to "tales and legends") significantly improves evaluation accuracy.
  • System Prompts Drive Personality: Evaluating models with only character descriptions (system prompts) vs. character memories (retrieved dialogues) revealed that descriptions are the primary driver of core personality traits.

Key Results

  • High Alignment: State-of-the-art RPAs exhibit personalities highly aligned with human perceptions, achieving a dimensional accuracy of up to 80.7% on the 16Personalities scale when evaluated using GPT-4.
  • Expert Rating Superiority: The Expert Rating (ER) method vastly outperformed traditional self-report baselines, showing higher consistency and a much stronger ability to capture distinct character personalities.
  • LLM as a Reliable Judge: In a validation study against human psychology annotators, GPT-4 achieved an 89.0% accuracy in simulating a human interviewer, making only 4% outright errors (mostly when the RPA provided contradictory answers).
Figure 4: Measured alignment (AccDim, %) of state-of- the-art RPAs on 14 scales.

Limitations and Future Work

While powerful, the InCharacter framework relies heavily on the capabilities of the interviewer LLM, meaning any inherent biases or parsing errors in models like GPT-4 could skew the personality scores. Furthermore, the evaluation treats character personalities as static. In reality, characters undergo development and change over time (e.g., James Bond across different eras), a dynamic aspect that current static benchmarks cannot capture. The authors point to modeling and evaluating these personality dynamics as a key direction for future research.

Summary updated

Abstract

Role-playing agents (RPAs), powered by large language models, have emerged as a flourishing field of applications. However, a key challenge lies in assessing whether RPAs accurately reproduce the personas of target characters, namely their character fidelity. Existing methods mainly focus on the knowledge and linguistic patterns of characters. This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales. Overcoming drawbacks of previous self-report assessments on RPAs, we propose InCharacter, namely Interviewing Character agents for personality tests. Experiments include various types of RPAs and LLMs, covering 32 distinct characters on 14 widely used psychological scales. The results validate the effectiveness of InCharacter in measuring RPA personalities. Then, with InCharacter, we show that state-of-the-art RPAs exhibit personalities highly aligned with the human-perceived personalities of the characters, achieving an accuracy up to 80.7%.
BibTeX
@inproceedings{wang2024incharacter,
  author = {Xintao Wang and Yunze Xiao and Jen-tse Huang and Siyu Yuan and Rui Xu and Haoran Guo and Quan Tu and Yaying Fei and Ziang Leng and Wei Wang and Jiangjie Chen and Cheng Li and Yanghua Xiao},
  title = {{InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.102},
}
Figure