TL;DR
This paper introduces a modular platform for creating believable, interactive AI characters that can engage in story-driven conversations. By integrating LLMs with multimodal sensing, expressive synthesis, and adaptive personality modeling, the system addresses key challenges in character consistency, emotion, and real-time responsiveness, exemplified by their "Digital Einstein" prototype.
What Problem Does This Paper Address?
Creating truly engaging and lifelike digital characters that can hold meaningful, story-driven conversations has been a long-standing challenge. While advancements in AI, particularly foundation models and prompt engineering, have enabled progress in individual areas like conversational AI, animation, and voice synthesis, combining these elements into a cohesive and believable interactive experience remains difficult. Existing systems often struggle with maintaining character integrity, managing personality and emotions, handling knowledge and memory, and achieving real-time synchronization across all components. This paper aims to bridge this gap by providing a unified and flexible platform that tackles these multifaceted challenges.
Methods
The core of the platform is a modular system built in Unity, featuring a digital character that transitions between idle, listening, thinking, and speaking states. Key components include:
- Conversational AI: Leverages both GPT-4o (for real-time engagement and thematic consistency) and a fine-tuned Llama 3 8B model (for local deployment and privacy). The system uses an embedding-based approach to track conversation topics and steer GPT-4o.
- Knowledge and Memory: A vector-based knowledge store maintains conversation history, retrieving the five most relevant snippets to inform responses.
- Personality and Emotion: Dynamic personality infusion, powered by GPT-4o, rewrites LLM responses to align with predefined personality profiles (vibrancy, conscientiousness, decency, artificiality, neuroticism). Emotions are extracted from responses using GPT-4o-mini to influence speech synthesis and facial animations.
- Speech Recognition and Synthesis: Utilizes Microsoft Azure Speech Services for real-time recognition and a Custom Neural Voice model for synthesized speech with emotional tone.
- Animation Synthesis: Facial animations are generated by NVIDIA Audio2Face, driven by the synthesized audio and emotion labels. Body animations are procedurally generated from a curated library of motion-capture clips.
- Visual Storytelling: Midjourney is used to generate topic-relevant images based on recent conversation turns, enhancing the visual experience.
- Real-World Interaction: A camera detects user presence, characteristics (age, gender), and attention, feeding this information into the system for situational awareness and personalized responses.
Datasets
- Synthetic Conversations: 71 synthetic human-Einstein conversations were generated for each of 62 curated topics, totaling 4,402 conversations. These were used for fine-tuning the Llama 3 model and creating topic embeddings.
- Motion Capture Data: Motion capture data from an actor performing various routines was used for animation.
- Speech Recordings: 1,618 recordings were used to fine-tune the custom neural voice model.
Key Insights and Ideas
- Modular Design is Crucial: The platform's modularity allows for easy swapping of components, making it adaptable to different characters and applications. This is a significant advantage over monolithic systems.
- Bridging the Gap Between Abstract Concepts and Embodiment: The system effectively links high-level AI outputs (LLM responses, personality traits) to concrete outputs like speech, facial animation, and body movement, creating a more cohesive experience.
- Physical Sliders for Intuitive Personality Control: Providing physical sliders for personality adjustment offers a more tangible and accessible way for users to influence the character's behavior compared to purely digital interfaces.
- Visual Storytelling Enhances Engagement: Dynamically generated images based on conversation topics can significantly enrich the user experience and reinforce learning or narrative elements.
- Balancing Real-time Performance and Quality: The use of dual LLMs (online GPT-4o and offline Llama 3) and careful optimization of each component demonstrates a practical approach to achieving real-time responsiveness without sacrificing conversational quality.
Key Results
- Real-time Performance: The system achieves cumulative runtimes of 4.7 seconds (GPT-4o) and 5.14 seconds (Llama 3), effectively masked by the avatar's thinking state.
- User Evaluation: GPT-4o demonstrated higher engagement in user evaluations at large international events, with more turns per session and longer responses compared to Llama 3. GPT-4o also covered a significantly higher number of topics.
- Topic Coverage: GPT-4o covered 49 and 42 topics at the tech and scientific events, respectively, showcasing its adaptability.
- Qualitative Improvements: The system successfully maintains character integrity, personality consistency, and story-driven interactions, as exemplified by the Digital Einstein prototype.
Limitations and Future Work
- Latency in Distributed Computing: The system can occasionally experience latency due to distributed computing, especially in fast-paced conversations.
- No Interruption Implementation: The ability for users to interrupt the character is not yet implemented but is planned for future work.
- Animation Synthesis Improvement: The authors aim to improve the animation synthesis model for more diverse and lively movements.
- Advanced Cognitive Modeling: Future work includes advancing cognitive modeling to better simulate human-like understanding and reasoning.
- Structured User Studies: The authors plan to conduct structured user studies to assess the contribution of individual system components to user engagement and perceived immersion.
Summary updated