LLMs vs. Chinese Anime Enthusiasts: A Comparative Study on Emotionally Supportive Role-Playing

Lanlan Qiu, Xiao Pu, Yeqi Feng, Tianxing HearXiv 2025 · DOI · arXiv

Introduction

Recent advancements in Large Language Models (LLMs) have demonstrated significant potential in both role-playing and emotional support. However, the intersection of these domains—Emotionally Supportive Role-Playing (ESRP)—remains under-explored. ESRP requires a model to provide empathetic guidance while strictly adhering to a specific character's persona, including their unique speech patterns, backstory, and worldview. This study introduces ChatAnime, a comprehensive dataset and evaluation framework designed to benchmark LLMs against human experts in providing persona-consistent emotional support.

Methodology

The researchers developed a rigorous pipeline to evaluate ESRP capabilities across various models:

  1. Scenario Design: Using GPT-4o, 60 real-world scenarios were generated, targeting four user personas (student, employee, freelancer, self-employed) across nine distinct emotional states to ensure a broad spectrum of supportive contexts.
  2. Character Grounding: To ensure factual accuracy and minimize hallucinations, character profiles were sourced from MoeGirl, a prominent Chinese ACG (Anime, Comics, and Games) wiki, providing models with detailed character profiles.
  3. Human Baseline: A group of 40 expert anime fans was selected from 300 applicants to provide high-quality, human-authored responses, ensuring deep character knowledge and creative writing skills.
  4. Multi-Dimensional Evaluation: Responses were assessed across nine fine-grained metrics categorized into Basic Dialogue (Consistency, Fluency, Coherency), Role-Playing (Knowledge, Speech Style, Behavioral Habit), and Emotional Support (Value, Experience Sharing, Demand Matching).

The ChatAnime Dataset

The ChatAnime dataset is a significant contribution to the field, comprising:

  • 20 Iconic Characters: Including popular figures like Luffy, Gintoki, Saber, and Taiga Aisaka.
  • 26,400 Dialogue Turns: Consisting of 2,400 human-written and 24,000 LLM-generated responses.
  • 132,480 Annotations: Detailed human ratings based on Likert scales, providing a robust foundation for evaluation.

Comparative Analysis and Key Results

The study revealed that top-tier LLMs have begun to rival or even surpass human fans in technical role-playing and emotional support metrics.

Comparison of emotional support responses from a human fan, DeepSeek-V3, and GPT-4.1 acting as Naruto.

Performance Highlights

  • DeepSeek-V3 emerged as the overall champion, outperforming human experts in both Role-Playing Capacity (3.95 vs. 3.79) and Emotional Support Capacity (4.13 vs. 3.93).
  • Character Knowledge: AI models outperformed humans in factual recall. By leveraging external knowledge bases, LLMs could reference obscure plot points more accurately than fans relying on memory, making "experience sharing" feel more authentic.
  • The Diversity Gap: Despite technical proficiency, LLMs struggle with linguistic variety. Humans achieved a diversity score of 3.90, whereas DeepSeek-V3 scored only 2.93. GPT-4o (referred to as 4.1 in the study) showed the best balance among AI models with a diversity score of 3.57.

Discussion: Capability vs. Creativity

A notable finding is the "Capability-Diversity Trade-off." There appears to be a negative correlation between a model's technical performance in persona maintenance and its linguistic creativity. High-performing models like DeepSeek-V3 often rely on repetitive catchphrases to maintain persona, whereas human fans provide more varied and nuanced dialogue.

Furthermore, the study distinguishes between Speech Style (mimicking tone) and Behavioral Habit (acting according to the character's values). Top-tier models are now reaching a point where they don't just "sound" like the character; they make decisions and offer advice that aligns with the character's established moral compass and personality.

Limitations and Future Directions

The current scope of the study is limited to Chinese-speaking communities and short-term (two-round) interactions. The authors acknowledge that maintaining character consistency over long-term, multi-session conversations remains a major challenge. Future research aims to expand into multimodal ESRP, incorporating visual and auditory elements to enhance the immersive experience of virtual companionship.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in role-playing conversations and providing emotional support as separate research directions. However, there remains a significant research gap in combining these capabilities to enable emotionally supportive interactions with virtual characters. To address this research gap, we focus on anime characters as a case study because of their well-defined personalities and large fan bases. This choice enables us to effectively evaluate how well LLMs can provide emotional support while maintaining specific character traits. We introduce ChatAnime, the first Emotionally Supportive Role-Playing (ESRP) dataset. We first thoughtfully select 20 top-tier characters from popular anime communities and design 60 emotion-centric real-world scenario questions. Then, we execute a nationwide selection process to identify 40 Chinese anime enthusiasts with profound knowledge of specific characters and extensive experience in role-playing. Next, we systematically collect two rounds of dialogue data from 10 LLMs and these 40 Chinese anime enthusiasts. To evaluate the ESRP performance of LLMs, we design a user experience-oriented evaluation system featuring 9 fine-grained metrics across three dimensions: basic dialogue, role-playing and emotional support, along with an overall metric for response diversity. In total, the dataset comprises 2,400 human-written and 24,000 LLM-generated answers, supported by over 132,000 human annotations. Experimental results show that top-performing LLMs surpass human fans in role-playing and emotional support, while humans still lead in response diversity. We hope this work can provide valuable resources and insights for future research on optimizing LLMs in ESRP. Our datasets are available at https://github.com/LanlanQiu/ChatAnime.
BibTeX
@misc{qiu2025llms,
  author = {Lanlan Qiu and Xiao Pu and Yeqi Feng and Tianxing He},
  title = {{LLMs vs. Chinese Anime Enthusiasts: A Comparative Study on Emotionally Supportive Role-Playing}},
  year = {2025},
  eprint = {2508.06388},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.06388},
}
Figure