Abstract
The demand for high-quality 3D avatar animation in film, gaming, and virtual broadcasting has surged, yet creating realistic, full-body movement remains a significant challenge. We present a production-ready system that synthesizes facial and body animations simultaneously from text and speech inputs. By employing a learning-based approach for facial expressions and a graph-based retrieval method for body motion, our system generates animations that are realistic, diverse, and semantically aligned with the input. This hybrid framework overcomes common pitfalls such as the "average pose" problem and modality isolation, making it suitable for real-time applications like virtual live broadcasting.
Introduction
The synthesis of lifelike 3D avatars is often considered a "holy grail" in digital media. Despite recent advancements, existing solutions frequently struggle with three core issues:
- Modality Isolation: Most systems treat facial and body animation as separate tasks, leading to a lack of coordination that triggers the "uncanny valley" effect.
- The "Average Pose" Problem: Purely generative deep learning models often struggle with the "one-to-many" mapping of speech to gesture. To minimize training error, these models tend to output a statistical average of all possible poses, resulting in dampened, "robotic" movements.
- Semantic Neglect: Many systems rely solely on audio phonetics, ignoring the underlying meaning of the text. This prevents avatars from performing specific gestures (e.g., pointing when saying "look there") that are essential for natural communication.
To address these challenges, we propose a dual-pathway framework developed by Alibaba’s XR Lab that leverages the strengths of both deep learning and motion retrieval.
Methodology
1. Facial Animation: Learning-Based Synthesis
Facial movements require high-frequency precision to maintain lip-sync accuracy and emotional nuance. We utilize a Multi-Pathway Transformer Network to handle two distinct sub-tasks:
- Lip Movement: A cross-modal transformer encoder processes audio features (MFCCs) alongside phoneme labels extracted from the text. To ensure phonetic accuracy, we implement an "articulation correction" step, which enforces proper mouth closure for bilabial consonants like /b/, /p/, and /m/.
- Expression Generation: Expressions are driven by two streams: "rhythmic" movements derived from speech audio and "intention-driven" movements derived from text sentiment analysis. During training, we employ Structural Similarity (SSIM) loss to ensure the generated facial parameters preserve the structural integrity of human expressions.
2. Body Animation: Graph-Based Retrieval
To avoid the "average pose" problem inherent in generative models, we adopt a Motion Graph approach for body animation. This method preserves the "sharpness" and variety of original human motion.
- Graph Construction: We segment high-quality Mo-Cap data into discrete clips based on motion strength minima. These clips form the nodes of a graph, where edges represent valid transitions.
- Retrieval and Optimization: The system searches the graph for an optimal path based on two criteria:
- Semantic Matching: Specific keywords (e.g., "Number one") trigger the retrieval of corresponding semantic gestures.
- Rhythmic Alignment: For non-semantic movements (such as weight shifting or general gesticulation), the system aligns the motion's tempo with the phonetic rhythm of the speech.
Data and Implementation
The system was trained and validated using two high-fidelity internal datasets:
- Facial Dataset: Consists of 3 hours of human talking data, captured as 3D facial parameters (52 ARKit-compatible blendshapes) and synchronized video.
- Body Motion Dataset: Collected via Vicon Mo-Cap with professional actors, featuring 24 categories of semantic motions and a wide array of non-semantic declarative actions.
Key Insights
A primary takeaway from this work is the effectiveness of a hybrid architecture. While deep learning excels at the subtle, high-frequency nuances of facial movement, graph-based retrieval is currently superior for body motion because it avoids the regression-to-the-mean effect. Furthermore, the inclusion of text-based sentiment analysis allows the system to capture "intentions" that audio signals alone might miss, such as a specific emphasis or a shift in mood.
Results and Conclusion
Our system produces animations that are qualitatively more expressive and diverse than traditional end-to-end deep learning models. By combining audio and phoneme data, we achieve highly accurate lip-syncing. The motion graph ensures that the avatar can synthesize multiple valid movement sequences for the same input, providing the diversity required for long-form content.
The framework is currently robust enough for real-time production environments, including Alibaba's Taobao virtual live broadcasts. Future work will focus on modeling individual personalities and enhancing the depth of emotional transitions to create even more engaging and interactive virtual personas.
Summary updated