Abstract
Due to the increasing demand in films and games, synthesizing 3D avatar animation has attracted much attention recently. In this work, we present a production-ready text/speech-driven full-body animation synthesis system. Given the text and corresponding speech, our system synthesizes face and body animations simultaneously, which are then skinned and rendered to obtain a video stream output. We adopt a learning-based approach for synthesizing facial animation and a graph-based approach to animate the body, which generates high-quality avatar animation efficiently and robustly. Our results demonstrate the generated avatar animations are realistic, diverse and highly text/speech-correlated.
Introduction
Generating high-quality 3D animations from speech is a classic "one-to-many" problem: for any given sentence, there are infinite ways a person might move their hands or change their expression. Previous research in this field often struggled with two primary issues: the "Average Pose" problem and semantic disconnect. Deep learning models trained to regress motion often predict the statistical mean of all possible movements, resulting in dampened or robotic animations. Furthermore, many systems focus only on the rhythm of speech (phonetic features) while ignoring the meaning of the words, leading to gestures that do not match the spoken intent.
Researchers from Alibaba’s XR Lab have developed a system that addresses these challenges by jointly synthesizing face and body movements using both audio signals and textual semantics.
System Architecture
The system utilizes a bifurcated pipeline that processes facial and body movements through specialized modules, ensuring both high-frequency facial detail and realistic body variety.
1. Face Animation (Learning-Based)
The facial module uses a multi-pathway transformer network to handle two distinct tasks:
- Lip Movement: A cross-modal transformer encoder processes audio features (MFCCs) and phoneme labels to predict 52 ARKit-standard blendshapes. An "articulation correction" step ensures precise mouth closure for specific phonemes like 'b', 'p', and 'm'.
- Expression Generation: This is split into rhythmic movements (following speech cadence) and intention-driven expressions. Sentiment analysis extracts semantic tags (e.g., "happiness") which are fused with rhythmic motion using an SSIM (Structural Similarity) loss function to maintain natural textures.
2. Body Animation (Graph-Based)
To avoid the "average pose" issue common in regression models, the authors opted for a Motion Graph approach.
- Construction: A library of professional Mo-Cap data was categorized into "semantic motions" (specific gestures like counting) and "non-semantic motions" (idling).
- Retrieval & Optimization: The system analyzes input text for keywords and searches the motion graph for a path that minimizes a cost function. This cost balances smooth transitions with phonetic rhythm and semantic requirements.
Datasets and Training
The system was trained on high-quality human talking data. Facial data included 3 hours of video and 3D facial parameters across diverse expressions. Body data was captured using a Vicon Mo-Cap system with professional actors, creating a database of 24 types of semantic actions and various non-semantic movements.
Key Insights and Results
- Semantic Integration: By using phoneme labels and sentiment analysis, the system achieves higher accuracy in lip-syncing and emotional resonance than audio-only systems.
- Hybrid Superiority: Deep learning excels at high-frequency facial movements, while graph-based retrieval preserves the variety and "sharpness" of real human body movement.
- Production Readiness: The system is currently deployed for Taobao virtual live broadcasts. Qualitative results show that avatars successfully perform semantic gestures (like counting) at the exact moment the corresponding words are spoken, outperforming previous joint-synthesis methods.
Conclusion
This framework demonstrates that combining learning-based facial synthesis with graph-based body retrieval creates a robust, production-ready pipeline for 3D avatars. Future work will focus on increasing personality diversity and enhancing real-time interactivity for live user engagement.
Summary updated