Abstract
Generating realistic, speech-driven 3D facial animations remains a challenge due to the "over-smoothed" and generic nature of existing emotional models. PESTalk introduces a novel framework that captures both fine-grained emotional cues and personalized expression styles. By combining a dual-stream audio extractor with a voiceprint-based style library, PESTalk allows digital avatars to express emotions in uniquely individualized ways, mirroring human anatomical and cultural diversity.
Introduction
Animating 3D faces directly from speech is essential for gaming, virtual reality, and digital avatars. While current models excel at lip-syncing, they often fail to convey realistic full-face emotional expressions. This failure stems from two primary limitations: the treatment of emotions as broad, universal categories and the reliance on audio encoders that neglect frequency-domain characteristics. These issues result in repetitive, stereotypical animations that lack the subtle nuances of individual human expression.
Methodology
PESTalk addresses these limitations through a pipeline designed to extract both semantic content and personalized emotional style from raw audio.
Core Components
- Dual-Stream Emotion Extractor (DSEE): DSEE utilizes parallel streams to capture both temporal variations (via TCN) and frequency-domain patterns (via a KAN-enhanced Conformer processing mel-spectrograms).
- Emotional Style Modeling Module (ESMM): This module extracts a voiceprint from a pre-trained speaker recognition model to build a
<speaker × emotion>library. This allows the model to retrieve personalized style embeddings in a zero-shot setting. - Partitioned Style-Guided Decoder: To mirror human biomechanics, the decoder is split into lower-face (high-frequency lip movements) and upper-face (low-frequency emotional expressions) components, preventing jittery artifacts.
Data and Evaluation
To train PESTalk, the authors developed 3D-EmoStyle, a large-scale dataset featuring over 180 subjects and 8 emotion categories. This was created by extracting pseudo-blendshape coefficients from massive 2D datasets (MEAD and HDTF) and mapping them to 52 FLAME head templates.
Key Results
PESTalk achieves state-of-the-art performance in both quantitative metrics and human perception. On the 3D-EmoStyle dataset, it reduced the Lip Blendshape Error (LBE) to 0.130 and the Mean Blendshape Error (MBE) to 0.277, significantly outperforming competitors like FaceDiffuser and EmoTalk.
In user studies, PESTalk received a Mean Opinion Score (MOS) of 4.349 for Personalized Emotional Expression, validating its ability to generate distinct facial animations for different speakers even when they express the same emotion.
Discussion and Insights
A critical insight of this work is that voiceprints serve as an effective proxy for facial animation styles, encoding both anatomical structures and vocal habits. Furthermore, the biomechanical decoupling of the face into upper and lower regions effectively prevents the interference between speech articulation and emotional expression.
While highly effective, the model is currently limited by its reliance on pseudo-labels generated from 2D-to-3D facial capture. Future improvements will likely depend on the acquisition of native 3D motion-capture data to further refine micro-expression realism.
Summary updated