Introduction
Speech-driven 3D facial animation has seen significant progress, particularly in the realm of lip-syncing. However, generating holistic facial expressions that feel truly "human" remains a persistent challenge. Most existing models suffer from the "uncanny valley" effect, where the mouth moves accurately to the audio, but the rest of the face—the eyes, brows, and cheeks—remains stiff or moves in a disjointed manner. This lack of coordination stems from treating facial parameters as independent variables rather than a physically and habitually correlated system.
CSTalk addresses this by introducing a framework that models the hidden spatial correlations between different facial regions. By supervising these correlations during the training process, the model generates animations where the entire face reacts cohesively to the emotional and phonetic content of speech.
Methodology: Correlation Supervision
The core innovation of CSTalk is its two-stage pipeline designed to enforce natural facial muscle coordination through a "Correlation Supervisor."
1. The Correlation Module
The researchers developed a Transformer-based encoder that treats facial control rigs as a sequence. Unlike standard temporal models, the self-attention mechanism here focuses on the relationships between different facial parts (e.g., how the "mouth dimple" rig relates to the "eyebrow raiser" rig). This module is pre-trained on high-quality animation data to recognize and classify emotional states based solely on these spatial movement patterns.
2. Animation Generation and Supervision
The generation architecture utilizes a Wav2vec 2.0 encoder to extract phonetic features from input audio. These features are then processed by a Temporal Convolutional Network (TCN) decoder to produce animation curves for 185 MetaHuman control rig parameters.
During training, the generated animation is passed through the pre-trained Correlation Module. If the generated facial movements do not exhibit the expected spatial correlations for the target emotion, the module penalizes the generator. This "Correlation Supervision" forces the model to learn the underlying structural "language" of facial expressions, ensuring that movements are coordinated across the entire face.
Dataset and Implementation
Because existing open-source datasets like VOCA provide vertex-based mesh data—which lacks the semantic control required for professional rigs—the authors curated a custom dataset using an iPhone’s Live Link Face app and Unreal Engine 5’s MetaHuman Animator plugin.
- Content: 500 samples (100 per emotion) across five states: neutral, angry, sad, surprised, and happy.
- Duration: Samples range from 4 to 9 seconds.
- Format: 185-parameter MetaHuman control rigs, making the output directly compatible with industry-standard game development and film pipelines.
Results and Discussion
CSTalk was evaluated against leading models, including Faceformer and Emotalk, using two key metrics: Lip Vertex Error (LVE) for phonetic accuracy and Emotional Vertex Error (EVE) for the quality of emotional expression.
| Metric | CSTalk | Emotalk |
|---|---|---|
| LVE (mm) | 2.538 | 2.954 |
| EVE (mm) | 2.084 | 2.875 |
Quantitative results show that CSTalk significantly outperforms previous methods in both categories. Qualitatively, the model produces vivid micro-expressions in the upper face (forehead and eyes) that sync naturally with the intensity of the speech. Ablation studies confirmed that the Correlation Module is the primary driver of this realism; removing it caused a significant spike in error, confirming that supervising the relationship between rigs is more vital for realism than simply increasing model capacity.
Conclusion
CSTalk represents a shift in 3D facial animation from focusing on temporal consistency to prioritizing spatial-structural consistency. By treating facial motion as a coordinated system of correlations, the framework moves closer to overcoming the uncanny valley. While the current model is tied to the MetaHuman rig architecture and a relatively small dataset, it provides a robust foundation for future work in zero-shot emotional transfer and high-fidelity digital human interaction.
Summary updated