Introduction
Recent advancements in Large Language Models (LLMs) have demonstrated significant potential in both role-playing and emotional support. However, the intersection of these domains—Emotionally Supportive Role-Playing (ESRP)—remains under-explored. ESRP requires a model to provide empathetic guidance while strictly adhering to a specific character's persona, including their unique speech patterns, backstory, and worldview. This study introduces ChatAnime, a comprehensive dataset and evaluation framework designed to benchmark LLMs against human experts in providing persona-consistent emotional support.
Methodology
The researchers developed a rigorous pipeline to evaluate ESRP capabilities across various models:
- Scenario Design: Using GPT-4o, 60 real-world scenarios were generated, targeting four user personas (student, employee, freelancer, self-employed) across nine distinct emotional states to ensure a broad spectrum of supportive contexts.
- Character Grounding: To ensure factual accuracy and minimize hallucinations, character profiles were sourced from MoeGirl, a prominent Chinese ACG (Anime, Comics, and Games) wiki, providing models with detailed character profiles.
- Human Baseline: A group of 40 expert anime fans was selected from 300 applicants to provide high-quality, human-authored responses, ensuring deep character knowledge and creative writing skills.
- Multi-Dimensional Evaluation: Responses were assessed across nine fine-grained metrics categorized into Basic Dialogue (Consistency, Fluency, Coherency), Role-Playing (Knowledge, Speech Style, Behavioral Habit), and Emotional Support (Value, Experience Sharing, Demand Matching).
The ChatAnime Dataset
The ChatAnime dataset is a significant contribution to the field, comprising:
- 20 Iconic Characters: Including popular figures like Luffy, Gintoki, Saber, and Taiga Aisaka.
- 26,400 Dialogue Turns: Consisting of 2,400 human-written and 24,000 LLM-generated responses.
- 132,480 Annotations: Detailed human ratings based on Likert scales, providing a robust foundation for evaluation.
Comparative Analysis and Key Results
The study revealed that top-tier LLMs have begun to rival or even surpass human fans in technical role-playing and emotional support metrics.
Performance Highlights
- DeepSeek-V3 emerged as the overall champion, outperforming human experts in both Role-Playing Capacity (3.95 vs. 3.79) and Emotional Support Capacity (4.13 vs. 3.93).
- Character Knowledge: AI models outperformed humans in factual recall. By leveraging external knowledge bases, LLMs could reference obscure plot points more accurately than fans relying on memory, making "experience sharing" feel more authentic.
- The Diversity Gap: Despite technical proficiency, LLMs struggle with linguistic variety. Humans achieved a diversity score of 3.90, whereas DeepSeek-V3 scored only 2.93. GPT-4o (referred to as 4.1 in the study) showed the best balance among AI models with a diversity score of 3.57.
Discussion: Capability vs. Creativity
A notable finding is the "Capability-Diversity Trade-off." There appears to be a negative correlation between a model's technical performance in persona maintenance and its linguistic creativity. High-performing models like DeepSeek-V3 often rely on repetitive catchphrases to maintain persona, whereas human fans provide more varied and nuanced dialogue.
Furthermore, the study distinguishes between Speech Style (mimicking tone) and Behavioral Habit (acting according to the character's values). Top-tier models are now reaching a point where they don't just "sound" like the character; they make decisions and offer advice that aligns with the character's established moral compass and personality.
Limitations and Future Directions
The current scope of the study is limited to Chinese-speaking communities and short-term (two-round) interactions. The authors acknowledge that maintaining character consistency over long-term, multi-session conversations remains a major challenge. Future research aims to expand into multimodal ESRP, incorporating visual and auditory elements to enhance the immersive experience of virtual companionship.
Summary updated