TL;DR
This paper introduces EmoVOCA, a novel approach and dataset for generating expressive 3D talking heads that blend speech-driven lip movements with emotional expressions. By developing a data-driven method to disentangle and then re-entangle speech and emotion facial deformations, they create a synthetic dataset that enables training more accurate and expressive 3D talking head generators than previous methods.
What Problem Does This Paper Address?
Generating realistic 3D talking heads from speech is a challenging task, especially when aiming to incorporate natural emotional expressions. A major bottleneck is the lack of comprehensive 3D datasets that combine diverse spoken sentences with a variety of facial expressions. Existing approaches often rely on fitting parametric 3D models (3DMMs) to 2D videos, which limits the precision of lip movements and overall expressiveness. While some methods attempt to overcome data scarcity by using 2D data as a proxy, they are fundamentally limited by the precision of the extracted 3D faces and the inherent constraints of parametric models. This paper tackles the problem of creating a richer 3D dataset and a method to generate expressive 3D talking heads that overcome these limitations.
Methods
The core of the proposed approach is a Double Encoder/Shared Decoder (DE-SD) architecture. This framework is designed to learn and combine speech and emotion facial deformations separately.
- Data Preparation: The method leverages two existing 3D face datasets: one with neutral talking heads (VOCAset) and another with expressive faces but no speech (Florence 4D). To prepare the data, the identity component is removed from each face by subtracting the neutral configuration. This results in displacement-based representations of speech (ST) and emotion (SE) motions.
- DE-SD Architecture:
- Two Encoders: One encoder (ET) learns speech-related facial movements from ST, and another (EE) learns expression-related deformations from SE. These encoders produce separate latent vectors.
- Shared Decoder: A single decoder (D) is trained to reconstruct the input displacements. During training, the latent vectors from both encoders are duplicated and concatenated to feed into the decoder. This allows the decoder to learn to reconstruct both speech and emotion motions.
- Inference: At inference time, the latent vectors from the speech and emotion encoders are concatenated. Coefficients (μt and μe) are used to control the contribution of speech and emotion features, allowing for a tunable blend of motions. The decoder then generates a combined displacement, which is added to a neutral face to produce the final expressive talking head.
- Dataset Creation: The DE-SD framework is used to generate a new synthetic dataset called EmoVOCA, which combines speech and emotional nuances.
Datasets
- VOCAset: A 3D talking head dataset with neutral expressions, used for learning speech-related movements.
- Florence 4D: A dataset of 3D expressive sequences without speech, used for learning emotion-related deformations.
- EmoVOCA (v1 & v2): A newly synthesized dataset created using the DE-SD framework. EmoVOCAv1 includes five emotions with three intensity levels, resulting in 7,200 sequences. EmoVOCAv2 expands this to 11 emotions with three intensity levels, totaling 15,840 sequences. This dataset is crucial for training expressive 3D talking head generators.
- VOCA-Test: Used for evaluating the generalization ability of the trained models.
Key Insights and Ideas
- Data-Driven Entanglement: The core idea is to learn speech and emotion deformations separately and then combine them in a data-driven manner, rather than relying on explicit disentanglement assumptions or parametric models.
- Spatial Disentanglement is Key: The paper highlights that speech and expression deformations often affect different facial regions. This spatial disentanglement allows the DE-SD architecture to effectively learn and combine these motions. The architecture leverages this by having encoders specialize and the decoder learn to activate different neurons based on the input features.
- Synthetic Data for Expressiveness: The creation of EmoVOCA is a significant contribution, addressing the critical lack of expressive 3D talking head datasets. This synthetic dataset enables training models that surpass the limitations of existing approaches.
- Compositional Ability: The DE-SD framework demonstrates strong compositional ability, allowing for the combination of various speech and emotion inputs to generate diverse outputs.
Key Results
- DE-SD Reconstruction Accuracy: The DE-SD model achieves good reconstruction accuracy on test sets of VOCAset and Florence 4D, with a weighted L2 loss improving results.
- Improved Generation Quality: Models trained on EmoVOCAv1/v2 (E-Faceformer and E-S2L+S2D) significantly outperform models trained on the original VOCAset and other state-of-the-art methods like EmoTalk and EMOTE in terms of both speech realism and emotional expression.
- Quantitative Metrics: E-S2L+S2D trained on EmoVOCAv2 achieved the lowest LVE (Lip Vertex Error) and UVE (Upper Vertex Error) on the VOCA-Test dataset, indicating superior lip synchronization and facial expression accuracy. For instance, E-S2L+S2D(v2) achieved an LVE of 1.845 mm and UVE of 0.501 mm.
- Qualitative Results: Visualizations show that the generated talking heads exhibit natural lip movements synchronized with speech and convincing emotional expressions, with the upper face being particularly influenced by the emotion.
- User Studies: User studies favored E-S2L+S2D trained on EmoVOCA, with participants preferring its outputs for speech realism and emotional expression over EmoTalk and EMOTE.
Limitations and Future Work
- Emotion Inconsistency: Some emotions were more difficult to inject accurately, as indicated by confusion matrices in user studies.
- Realism Gaps: The current framework lacks some realism in areas like eye blinking and head pose.
- Future Directions: The authors suggest investigating methods to improve eye blinking and head pose control, as well as exploring ways to better inject certain emotions. They also note that the DE-SD framework's flexibility allows for generating numerous additional versions of the EmoVOCA dataset by varying the emotional inputs.
Summary updated