Abstract
CharacterCraft is a novel framework designed to enhance role-playing agents (RPAs) by addressing the "literature-reality gap"—the discrepancy between how characters speak in novels and how real users interact with AI. By integrating a high-precision dialogue extraction pipeline, an iterative "augmentation-reconstruction" (A-R) method, and a context-aware memory module, CharacterCraft produces high-quality Chinese datasets that significantly improve character consistency and conversational naturalness. Experimental results demonstrate that CharacterCraft-enhanced models outperform existing benchmarks and even surpass state-of-the-art models like GPT-4o in specific role-playing dimensions.
Introduction
The rise of role-playing agents (RPAs) on platforms like Character.AI has highlighted a significant challenge: the "Literature-Reality Gap." While high-quality training data is often sourced from novels, characters in literature speak to advance plots, whereas users interact with AI for information or specific role-play scenarios. This discrepancy leads to agents that struggle with common user queries. Additionally, many synthetic datasets suffer from "trait dilution," where responses lack the unique linguistic flavor and emotional depth of the original characters.
CharacterCraft addresses these issues by moving beyond static character profiles (name, age, personality) to capture "dynamic" traits—how characters make decisions and express emotions in high-stakes, specific scenes.
The CharacterCraft Framework
The CharacterCraft framework is built upon three core pillars designed to refine data collection and enhance agent performance.
1. High-Precision Extraction
Accurately identifying "who said what" in complex Chinese novels is a non-trivial task. The authors found that even general-purpose models like GPT-4o struggle with this. To solve this, they fine-tuned a specialized model, Qwen2-1.5B-Instruct, which achieved a state-of-the-art 94.94% accuracy in dialogue extraction, outperforming much larger models.
2. Iterative Augmentation-Reconstruction (A-R)
To bridge the literature-reality gap, CharacterCraft employs a two-step A-R process:
- Augmentation: An LLM generates a contextual explanation for a specific book dialogue.
- Reconstruction: The original character's query is masked, and the LLM "reconstructs" a new query that mirrors common user language while maintaining the original intent. This ensures the agent learns to respond to realistic user inputs without losing character identity.
3. Context-Aware Memory Retrieval (CMR)
Rather than relying solely on a static bio, CharacterCraft utilizes Retrieval-Augmented Generation (RAG). It stores thousands of character-specific scenes as "memories" and retrieves the most relevant ones in real-time to guide the agent's responses, ensuring consistency in decision-making and emotional expression.
CharacterCraft-Data
The framework was used to create CharacterCraft-Data, a large-scale Chinese role-playing dataset. It features 21,392 multi-turn dialogues and 121,418 utterances across 369 unique characters from 15 diverse Chinese novels.
As shown in the comparison table, CharacterCraft-Data is significantly larger than previous efforts and exhibits lower Kullback-Leibler (KL) divergence from real human conversation (using the NaturalConv dataset as a baseline), indicating a higher degree of naturalness.
Evaluation and Results
The authors introduced a Reference-Guided Evaluation method. Standard "LLM-as-a-judge" approaches often fail because the judge lacks deep knowledge of the character. By providing the judge with actual dialogue from the source novel as a ground-truth reference, the evaluation aligns much more closely with human judgment (61% Kendall correlation vs. 42% for reference-free methods).
Key findings include:
- Performance: Models enhanced with the CMR module (e.g., Qwen2.5-72B+CMR) achieved superior scores, even surpassing GPT-4o in character consistency and decision-making.
- Dynamic Traits: While most models handle static traits (style, background) well, the CMR module is crucial for capturing dynamic traits like emotion and decision-making.
- Robustness: The study found that standard reward models are brittle; minor punctuation changes can significantly impact consistency scores, highlighting the need for more robust evaluation frameworks like the one proposed.
Conclusion and Future Work
CharacterCraft successfully bridges the gap between literary data and practical AI interaction. While currently focused on Chinese literature, the methodology is language-agnostic. Future work includes expanding the framework to support multilingual RPAs and incorporating real-world personas such as historical figures or celebrities to broaden the application scope in education and professional training.
Summary updated