CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation

Xiangyu Liang, Wenlin Zhuang, Tianyong Wang, Guangxing Geng, Guangyue Geng, Haifeng Xia, Siyu Xia

Introduction

Speech-driven 3D facial animation has seen significant progress, particularly in the realm of lip-syncing. However, generating holistic facial expressions that feel truly "human" remains a persistent challenge. Most existing models suffer from the "uncanny valley" effect, where the mouth moves accurately to the audio, but the rest of the face—the eyes, brows, and cheeks—remains stiff or moves in a disjointed manner. This lack of coordination stems from treating facial parameters as independent variables rather than a physically and habitually correlated system.

CSTalk addresses this by introducing a framework that models the hidden spatial correlations between different facial regions. By supervising these correlations during the training process, the model generates animations where the entire face reacts cohesively to the emotional and phonetic content of speech.

Methodology: Correlation Supervision

The core innovation of CSTalk is its two-stage pipeline designed to enforce natural facial muscle coordination through a "Correlation Supervisor."

1. The Correlation Module

The researchers developed a Transformer-based encoder that treats facial control rigs as a sequence. Unlike standard temporal models, the self-attention mechanism here focuses on the relationships between different facial parts (e.g., how the "mouth dimple" rig relates to the "eyebrow raiser" rig). This module is pre-trained on high-quality animation data to recognize and classify emotional states based solely on these spatial movement patterns.

2. Animation Generation and Supervision

The generation architecture utilizes a Wav2vec 2.0 encoder to extract phonetic features from input audio. These features are then processed by a Temporal Convolutional Network (TCN) decoder to produce animation curves for 185 MetaHuman control rig parameters.

During training, the generated animation is passed through the pre-trained Correlation Module. If the generated facial movements do not exhibit the expected spatial correlations for the target emotion, the module penalizes the generator. This "Correlation Supervision" forces the model to learn the underlying structural "language" of facial expressions, ensuring that movements are coordinated across the entire face.

Dataset and Implementation

Because existing open-source datasets like VOCA provide vertex-based mesh data—which lacks the semantic control required for professional rigs—the authors curated a custom dataset using an iPhone’s Live Link Face app and Unreal Engine 5’s MetaHuman Animator plugin.

  • Content: 500 samples (100 per emotion) across five states: neutral, angry, sad, surprised, and happy.
  • Duration: Samples range from 4 to 9 seconds.
  • Format: 185-parameter MetaHuman control rigs, making the output directly compatible with industry-standard game development and film pipelines.

Results and Discussion

CSTalk was evaluated against leading models, including Faceformer and Emotalk, using two key metrics: Lip Vertex Error (LVE) for phonetic accuracy and Emotional Vertex Error (EVE) for the quality of emotional expression.

Metric CSTalk Emotalk
LVE (mm) 2.538 2.954
EVE (mm) 2.084 2.875

Quantitative results show that CSTalk significantly outperforms previous methods in both categories. Qualitatively, the model produces vivid micro-expressions in the upper face (forehead and eyes) that sync naturally with the intensity of the speech. Ablation studies confirmed that the Correlation Module is the primary driver of this realism; removing it caused a significant spike in error, confirming that supervising the relationship between rigs is more vital for realism than simply increasing model capacity.

Conclusion

CSTalk represents a shift in 3D facial animation from focusing on temporal consistency to prioritizing spatial-structural consistency. By treating facial motion as a coordinated system of correlations, the framework moves closer to overcoming the uncanny valley. While the current model is tied to the MetaHuman rig architecture and a relatively small dataset, it provides a robust foundation for future work in zero-shot emotional transfer and high-fidelity digital human interaction.

Summary updated

Abstract

Speech-driven 3D facial animation technology has been developed for years, but its practical application still lacks expectations. The main challenges lie in data limitations, lip alignment, and the naturalness of facial expressions. Although lip alignment has seen many related studies, existing methods struggle to synthesize natural and realistic expressions, resulting in a mechanical and stiff appearance of facial animations. Even with some research extracting emotional features from speech, the randomness of facial movements limits the effective expression of emotions. To address this issue, this paper proposes a method called CSTalk (Correlation Supervised) that models the correlations among different regions of facial movements and supervises the training of the generative model to generate realistic expressions that conform to human facial motion patterns. To generate more intricate animations, we employ a rich set of control parameters based on the metahuman character model and capture a dataset for five different emotions. We train a generative network using an autoencoder structure and input an emotion embedding vector to achieve the generation of user-control expressions. Experimental results demonstrate that our method outperforms existing state-of-the-art methods.
BibTeX
@inproceedings{liang2024cstalk,
  author = {Xiangyu Liang and Wenlin Zhuang and Tianyong Wang and Guangxing Geng and Guangyue Geng and Haifeng Xia and Siyu Xia},
  title = {{CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation}},
  booktitle = {FG},
  year = {2024},
  url = {https://openreview.net/forum?id=4BBMtn1xV1},
}
Figure