A Platform for Interactive AI Character Experiences

Rafael Wampfler, Chen Yang, Dillon Elste, Nikola Kovacevic, Philine Witzig, Markus Gross

TL;DR

This paper introduces a modular platform for creating believable, interactive AI characters that can engage in story-driven conversations. By integrating LLMs with multimodal sensing, expressive synthesis, and adaptive personality modeling, the system addresses key challenges in character consistency, emotion, and real-time responsiveness, exemplified by their "Digital Einstein" prototype.

What Problem Does This Paper Address?

Creating truly engaging and lifelike digital characters that can hold meaningful, story-driven conversations has been a long-standing challenge. While advancements in AI, particularly foundation models and prompt engineering, have enabled progress in individual areas like conversational AI, animation, and voice synthesis, combining these elements into a cohesive and believable interactive experience remains difficult. Existing systems often struggle with maintaining character integrity, managing personality and emotions, handling knowledge and memory, and achieving real-time synchronization across all components. This paper aims to bridge this gap by providing a unified and flexible platform that tackles these multifaceted challenges.

Methods

The core of the platform is a modular system built in Unity, featuring a digital character that transitions between idle, listening, thinking, and speaking states. Key components include:

  • Conversational AI: Leverages both GPT-4o (for real-time engagement and thematic consistency) and a fine-tuned Llama 3 8B model (for local deployment and privacy). The system uses an embedding-based approach to track conversation topics and steer GPT-4o.
  • Knowledge and Memory: A vector-based knowledge store maintains conversation history, retrieving the five most relevant snippets to inform responses.
  • Personality and Emotion: Dynamic personality infusion, powered by GPT-4o, rewrites LLM responses to align with predefined personality profiles (vibrancy, conscientiousness, decency, artificiality, neuroticism). Emotions are extracted from responses using GPT-4o-mini to influence speech synthesis and facial animations.
  • Speech Recognition and Synthesis: Utilizes Microsoft Azure Speech Services for real-time recognition and a Custom Neural Voice model for synthesized speech with emotional tone.
  • Animation Synthesis: Facial animations are generated by NVIDIA Audio2Face, driven by the synthesized audio and emotion labels. Body animations are procedurally generated from a curated library of motion-capture clips.
  • Visual Storytelling: Midjourney is used to generate topic-relevant images based on recent conversation turns, enhancing the visual experience.
  • Real-World Interaction: A camera detects user presence, characteristics (age, gender), and attention, feeding this information into the system for situational awareness and personalized responses.

Datasets

  • Synthetic Conversations: 71 synthetic human-Einstein conversations were generated for each of 62 curated topics, totaling 4,402 conversations. These were used for fine-tuning the Llama 3 model and creating topic embeddings.
  • Motion Capture Data: Motion capture data from an actor performing various routines was used for animation.
  • Speech Recordings: 1,618 recordings were used to fine-tune the custom neural voice model.

Key Insights and Ideas

  • Modular Design is Crucial: The platform's modularity allows for easy swapping of components, making it adaptable to different characters and applications. This is a significant advantage over monolithic systems.
  • Bridging the Gap Between Abstract Concepts and Embodiment: The system effectively links high-level AI outputs (LLM responses, personality traits) to concrete outputs like speech, facial animation, and body movement, creating a more cohesive experience.
  • Physical Sliders for Intuitive Personality Control: Providing physical sliders for personality adjustment offers a more tangible and accessible way for users to influence the character's behavior compared to purely digital interfaces.
  • Visual Storytelling Enhances Engagement: Dynamically generated images based on conversation topics can significantly enrich the user experience and reinforce learning or narrative elements.
  • Balancing Real-time Performance and Quality: The use of dual LLMs (online GPT-4o and offline Llama 3) and careful optimization of each component demonstrates a practical approach to achieving real-time responsiveness without sacrificing conversational quality.

Key Results

  • Real-time Performance: The system achieves cumulative runtimes of 4.7 seconds (GPT-4o) and 5.14 seconds (Llama 3), effectively masked by the avatar's thinking state.
  • User Evaluation: GPT-4o demonstrated higher engagement in user evaluations at large international events, with more turns per session and longer responses compared to Llama 3. GPT-4o also covered a significantly higher number of topics.
  • Topic Coverage: GPT-4o covered 49 and 42 topics at the tech and scientific events, respectively, showcasing its adaptability.
  • Qualitative Improvements: The system successfully maintains character integrity, personality consistency, and story-driven interactions, as exemplified by the Digital Einstein prototype.

Limitations and Future Work

  • Latency in Distributed Computing: The system can occasionally experience latency due to distributed computing, especially in fast-paced conversations.
  • No Interruption Implementation: The ability for users to interrupt the character is not yet implemented but is planned for future work.
  • Animation Synthesis Improvement: The authors aim to improve the animation synthesis model for more diverse and lively movements.
  • Advanced Cognitive Modeling: Future work includes advancing cognitive modeling to better simulate human-like understanding and reasoning.
  • Structured User Studies: The authors plan to conduct structured user studies to assess the contribution of individual system components to user engagement and perceived immersion.

Summary updated

BibTeX
@inproceedings{gross2025platform,
  author = {Markus Gross and Rafael Wampfler and Chen Yang and Dillon Elste and Nikola Kovacevic and Philine Witzig and Markus Gross},
  title = {{A Platform for Interactive AI Character Experiences}},
  booktitle = {Siggraph (conference Paper Track)},
  year = {2025},
  doi = {10.1145/3721238.3730762},
}
Figure