Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in role-playing, yet evaluating these "persona" abilities remains a significant challenge. Traditional benchmarks often rely on static methods such as multiple-choice questions, which test knowledge about a character rather than the ability to perform as one, or single-turn dialogue completion, which fails to capture how a model's performance drifts over a long conversation. To address these limitations, the authors propose DMT-RoleBench, a framework designed to assess LLMs through dynamic, multi-turn dialogues that mirror real-world user interactions.
The DMT-RoleBench Framework
The core of DMT-RoleBench is a three-part system designed to simulate human-like evaluation flows:
- Dynamic Dialogue Generator (DDG): This component uses a "Topic Manager" and an "Intent Manager" (powered by a state-of-the-art model like GPT-4o) to act as an interviewer. Unlike fixed scripts, the DDG generates queries dynamically based on conversation history, testing specific "evaluative intents" such as character consistency and backstory knowledge.
- DMT-RM (The Judge): To move beyond subjective 1-10 scales, the researchers fine-tuned a Qwen2-7B model on high-quality human annotations. This judge provides binary "good/bad" labels across 14 different sub-metrics.
- DMT-Score: A novel scoring algorithm that employs a "strict" approach. If a model breaks character or fails in a single turn, all subsequent turns in that session are penalized. This mirrors real-world user behavior, where a single immersion-breaking response can terminate the experience.
Dataset and Evaluation Setup
The benchmark utilizes the DMT-RoleBench Seed Dataset, comprising 996 samples across six role categories: Fictional Characters, Historical Figures, Professional Occupations, Emotional Companionships, Utility Assistants, and Interactive Game NPCs. Evaluation is conducted using three prompt formats—Minimalist (name only), Overview (brief bio), and Detailed (comprehensive instructions)—and seven evaluation intents, such as "Identity Recognition" and "Knowledge Boundary."
Results and Analysis
The evaluation of seven major models (including GPT-4o, Qwen2, and Doubao) revealed several key insights:
- Top Performers: Doubao emerged as the strongest overall model, particularly excelling in "Role Embodying" and "Human Likeness."
- Cultural Nuance: Local Chinese models generally outperformed the GPT series on roles rooted in Chinese culture, likely due to better alignment with specific cultural source materials.
- Performance Decay: A critical finding was the consistent downward trend in quality as dialogue turns increased. While models like Doubao and GPT-4o remained relatively stable, others like GPT-3.5 showed significant decline over time.
- Prompt Robustness: Models like Doubao and ErnieBot showed high consistency even with minimalist prompts, indicating robust internal character knowledge graphs.
- The "As an AI" Trap: Even high-performing models frequently slipped into "As an AI assistant..." language, which is a significant failure mode for immersive role-playing.
Conclusion
DMT-RoleBench highlights the challenges of maintaining immersive personas in long-form dialogue. While current models excel in short interactions, performance decay and character drift remain significant hurdles, especially in complex scenarios like game interaction roles. Future work will focus on expanding the benchmark to global cultural contexts and improving agent performance in procedural narrative steering.
Reference: Yuan, D., et al. (2025). DMT-RoleBench: A Dynamic Multi-Turn Dialogue Based Benchmark for Role-Playing Evaluation of Large Language Model and Agent. Proceedings of the 39th AAAI Conference on Artificial Intelligence.
Summary updated