Abstract
Recent advancements in Large Language Models (LLMs) have shown outstanding potential for role-playing applications. Evaluating these capabilities is becoming crucial yet remains challenging. Existing benchmarks mostly adopt a character-centric approach, simplify user-character interactions to isolated Q&A tasks, and fail to reflect real-world applications. To address this limitation, we introduce RMTBench, a comprehensive user-centric bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. RMTBench includes custom characters with detailed backgrounds and abstract characters defined by simple traits, enabling evaluation across various user scenarios. Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications. Furthermore, we construct an authentic multi-turn dialogue simulation mechanism. With carefully selected evaluation dimensions and LLM-based scoring, this mechanism captures the complex intention of conversations between the user and the character. By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements, offering a more effective framework for assessing role-playing capabilities in LLMs.
Introduction
The evaluation of role-playing in Large Language Models (LLMs) has traditionally focused on "character-centric" metrics. These benchmarks often treat role-playing as a trivia exercise, testing the model's knowledge of a character's biography through isolated Q&A. However, real-world usage—ranging from emotional support to collaborative storytelling—requires a "user-centric" approach where the model navigates dynamic, multi-turn conversations while maintaining a consistent persona.
RMTBench addresses this gap by prioritizing user intentions over static character backgrounds. By simulating over 8,000 rounds of dialogue across 80 diverse characters, it provides a more authentic assessment of how models perform in practical deployment scenarios.
Methodology
RMTBench introduces a bilingual (English and Chinese) dataset structured around five distinct interaction scenarios designed to test the limits of persona consistency and user satisfaction:
- Character Understanding: Probing specific knowledge and history related to the character.
- Character Maintenance: Testing the model's ability to stay in character when challenged about its AI nature (e.g., "Are you ChatGPT?").
- Implicit User Intentions: Responding to user needs, such as seeking advice, within the character's unique persona.
- User Preference Awareness: Reasoning based on personal information shared by the user earlier in the conversation.
- Sensitive User Behavior: Maintaining authenticity and safety when faced with insults or inappropriate requests.
Performance is measured across seven dimensions, including Emotional Expression (EE), Plot Advancement (PA), and User Preference Awareness (UPA). Evaluation is conducted using an "LLM-as-a-judge" framework (specifically Qwen2.5-72B-Instruct), which has been validated against human annotations to ensure high reliability.
Key Insights and Results
The benchmark reveals several critical findings regarding current LLM capabilities in role-playing contexts:
The Multi-Turn Performance Drop
A significant discovery is the "Pseudo Multi-Turn" trap. Models evaluated on pre-written, high-quality histories often appear more capable than they are. In authentic multi-turn setups where models must generate their own history, performance drops by an average of 4 points, highlighting the difficulty of maintaining consistency over time.
Character Maintenance Challenges
The "Character Maintenance" (CM) dimension proved to be a significant filter. Even advanced models occasionally "leak" their AI identity when pushed, indicating that persona consistency is a distinct skill separate from general reasoning or knowledge retrieval.
Model Rankings
Closed-source models generally lead the field. Qwen2.5-Max achieved the highest average scores (81.4 in English, 82.9 in Chinese), followed by GPT-4o and Claude 3.5 Sonnet. Among open-source models, Llama-3.3-70B showed strong competitiveness, particularly in English. However, smaller "Instruct" versions (e.g., Qwen2.5-7B) struggle significantly with complex tasks like User Preference Awareness, often scoring below 50%.
Conclusion
RMTBench shifts the paradigm of role-playing evaluation from knowledge retrieval to intention fulfillment. By highlighting the performance decay in long-context, multi-turn dialogues and the difficulty of maintaining character integrity, it provides a roadmap for developing more robust and engaging role-playing AI. The dataset and evaluation framework are released to the community to foster further research in user-centric interaction.
Summary updated