TL;DR
This paper introduces RMTBench, a novel user-centric benchmark for evaluating Large Language Models (LLMs) in role-playing scenarios. By shifting focus from character fidelity to user intention fulfillment in multi-turn dialogues, RMTBench offers a more realistic assessment of LLMs' conversational abilities for practical applications.
What Problem Does This Paper Address?
The burgeoning popularity of LLMs in role-playing applications, exemplified by platforms like Character.AI, highlights the need for robust evaluation. However, existing benchmarks often fall short by adopting a "character-centric" approach. This means they tend to focus on how well an LLM can mimic a specific character's traits and background, often through isolated Q&A tasks. This overlooks the crucial aspect of why users engage in role-playing: to fulfill specific intentions, whether for entertainment, emotional support, or knowledge acquisition. These character-centric methods fail to capture the dynamic, multi-turn nature of real-world user interactions and create a significant gap between academic evaluation and practical deployment.
Methods
The core of RMTBench is its user-centric dialogue construction. Instead of relying solely on character descriptions, dialogues are built around explicit user motivations and goals. The benchmark incorporates five distinct evaluation scenarios:
- Character Understanding: Assesses if the LLM can grasp and express character-specific knowledge and background.
- Character Maintenance: Evaluates the LLM's ability to maintain character persona, even when faced with "AI identity" probing questions.
- Implicit User Intentions Response: Focuses on how well the LLM identifies and responds to user goals, providing character-appropriate advice or interaction.
- User Preference Awareness and Reasoning: Tests the LLM's capacity to remember and utilize user preferences expressed over multiple turns.
- Sensitive User Behavior Handling: Examines how LLMs manage potentially inappropriate or sensitive user inputs, balancing character authenticity with ethical appropriateness.
For evaluation, RMTBench uses seven dimensions: Emotional Expression (EE), Emotional Comprehension (EC), Plot Advancement (PA), Character Understanding (CU), Character Maintenance (CM), Security (SEC), and User Preference Awareness (UPA). Both human annotators and an LLM (Qwen2.5-72B-Instruct) were used for scoring.
Datasets
RMTBench is a newly introduced benchmark. It comprises:
- 80 diverse characters: Including celebrities, fictional characters, and custom characters (both abstract and with detailed backgrounds).
- Over 8,000 dialogue rounds: Spanning both English and Chinese.
- User-centric dialogue construction: Dialogues are generated using Claude 3.5 Sonnet, focusing on user intentions rather than predefined character responses.
- Extra-long dialogues: Exceeding 20 rounds to simulate sustained interactions.
Key Insights and Ideas
- User intention is paramount: The study strongly suggests that evaluating LLMs based on their ability to fulfill user intentions, rather than just character consistency, is crucial for real-world applicability.
- Multi-turn evaluation is essential: Pseudo multi-turn evaluations (using pre-set responses) tend to overestimate LLM performance, highlighting the need for authentic, end-to-end multi-turn dialogue generation and evaluation.
- Dialogue length matters: Performance can degrade in longer dialogues, especially for open-source models, indicating challenges in maintaining context and character consistency over extended interactions.
- Character Maintenance is a key challenge: Even competitive closed-source models sometimes struggle to avoid revealing their AI identity, underscoring the difficulty of seamless role-playing.
Key Results
- Closed-source models generally outperform open-source models: ChatGPT-40-Latest and Claude 3.5 Sonnet achieved higher average scores across dimensions in both English and Chinese.
- Qwen2.5-Max showed strong performance: It led in most dimensions for both languages among open-source models, maintaining a competitive edge.
- Language performance varies for open-source models: Some open-source models exhibited unstable performance across English and Chinese, unlike the more consistent closed-source options.
- Character Maintenance (CM) was the most challenging dimension: Models, including strong performers, showed the highest standard deviation and range in this area, indicating a common failure point.
Limitations and Future Work
The authors acknowledge several limitations:
- Nuance in dialogue generation: Automatically generated dialogues might not fully capture the subtle complexities of user intentions and role-playing scenarios.
- Annotator correlation: The correlation scores between automated annotators and human annotators were not exceptionally high.
- Sensitive data handling: While ethically constructed and safeguarded, the dataset includes sensitive content to evaluate safety boundaries, requiring careful access control.
Future work could involve further refining the dialogue generation process to capture deeper nuances and exploring more sophisticated evaluation metrics. The release of the RMTBench code and datasets is planned to facilitate further research in this area.
Summary updated