ESGaussianFace: Emotional and Stylized Audio-Driven Facial Animation via 3D Gaussian Splatting

Chuhang Ma, Shuai Tan, Ye Pan, Jiaolong Yang, Xin Tong

TL;DR

ESGaussianFace is a novel framework that generates high-fidelity, 3D-consistent talking head videos that are both emotionally expressive and artistically stylized. By leveraging 3D Gaussian Splatting and a specialized spatial attention mechanism, it achieves real-time performance (~70 FPS) while outperforming previous state-of-the-art methods in lip-sync accuracy and emotional expressiveness.

What Problem Does This Paper Address?

Audio-Driven Facial Animation (ADFA) aims to create realistic videos of a person speaking based solely on an audio clip. While the field has made strides in "neutral" talking heads, two major hurdles remain:

  1. Emotional Realism: Most models struggle to convey complex emotions (angry, happy, surprised) accurately while maintaining precise lip-sync.
  2. Stylization & 3D Consistency: 2D-based methods often produce "jittery" results or distortions when the head moves. Conversely, 3D methods like Neural Radiance Fields (NeRF) are often too slow for real-time use or require training a separate model for every single emotion.

The authors tackle the challenge of creating a single, efficient model that can handle diverse emotions and artistic styles (like cartoons or oil paintings) without losing 3D structural integrity.

Methods

The core of ESGaussianFace is built on 3D Gaussian Splatting (3DGS), an explicit 3D representation that allows for much faster rendering than implicit methods like NeRF. The architecture consists of three main pillars:

  • Triplane-based 3D Gaussian Generator: It uses a multi-resolution triplane to encode the spatial features of a "canonical" (neutral) face, which serves as the base for all animations.
  • Emotion-audio-guided Spatial Attention Module (ESAM): This is the "brain" of the fusion process. It uses cross-attention to ensure that audio features primarily influence the mouth region, while emotion features (extracted via 3DMM coefficients) influence broader regions like the eyes and eyebrows.
  • ESGaussian Deformation Prediction Module: This module predicts how the 3D Gaussian points should shift, scale, or rotate to match the target emotion and art style. It uses two distinct predictors: one for emotional movement ($P_{emo}$) and one for stylistic changes ($P_{sty}$).

To make training stable, the authors use a multi-stage training strategy:

  1. Neutral Stage: The model learns basic lip-syncing.
  2. Emotion Stage: The model learns how Gaussians deform to express feelings.
  3. Stylization Stage: The model learns to apply artistic textures and shapes.

Datasets

The researchers evaluated their model using several high-quality benchmarks:

  • MEAD: A large-scale dataset featuring 60 actors performing 8 different emotions at 3 intensity levels.
  • RAVDESS: An emotional speech and song dataset with 24 actors.
  • Art Datasets: Various artistic references were used to train the stylization component, with VToonify providing the "ground truth" for stylized targets during the training phase.

Key Insights and Ideas

  • Spatial Awareness is Key: By using spatial attention (ESAM), the model "knows" that audio shouldn't affect the forehead and that "happiness" involves more than just the mouth—it affects the squint of the eyes and the lift of the cheeks.
  • Decomposition of Motion: Instead of trying to learn "stylized-emotional-speech" all at once, the authors decoupled the tasks. Learning lip-sync first provides a structural foundation that prevents the model from collapsing when complex emotions or artistic distortions are added later.
  • Efficiency via 3DGS: Moving from NeRF to 3DGS wasn't just about speed; the explicit nature of Gaussian points makes it easier to apply precise deformations controlled by external features like style encodings.

Key Results

  • Quantitative Superiority: ESGaussianFace achieved the highest scores in PSNR (image quality) and LMD (Landmark Distance, measuring expression accuracy) compared to SOTA models like GaussianTalker and Style2Talker.
  • Real-time Performance: The model runs at 69.6 FPS, making it suitable for interactive applications, whereas many NeRF-based competitors struggle to reach 1 FPS.
  • 3D Consistency: Unlike 2D GAN-based methods, ESGaussianFace allows for smooth camera rotations (multi-view generation) without the facial features "sliding" or warping unnaturally.

Limitations and Future Work

  • Emotion Intensity: The model is somewhat limited by the training data (MEAD only has three intensity levels). It can struggle to generate very subtle or extremely nuanced emotional transitions that fall between these levels.
  • Character Specificity: Currently, a separate 3D Gaussian field must be trained for each specific character identity. The authors suggest that future work could explore "one-shot" or generalized models that can animate any face without per-person retraining.
  • Stylization Sources: The current stylization relies on videos processed by VToonify. Integrating more advanced diffusion-based stylization methods could lead to even more diverse artistic outputs.

Summary updated

Abstract

Most current audio-driven facial animation research primarily focuses on generating videos with neutral emotions. While some studies have addressed the generation of facial videos driven by emotional audio, efficiently generating high-quality talking head videos that integrate both emotional expressions and style features remains a significant challenge. In this paper, we propose ESGaussianFace, an innovative framework for emotional and stylized audio-driven facial animation. Our approach leverages 3D Gaussian Splatting to reconstruct 3D scenes and render videos, ensuring efficient generation of 3D consistent results. We propose an emotion-audio-guided spatial attention method that effectively integrates emotion features with audio content features. Through emotion-guided attention, the model is able to reconstruct facial details across different emotional states more accurately. To achieve emotional and stylized deformations of the 3D Gaussian points through emotion and style features, we introduce two 3D Gaussian deformation predictors. Futhermore, we propose a multi-stage training strategy, enabling the step-by-step learning of the character's lip movements, emotional variations, and style features. Our generated results exhibit high efficiency, high quality, and 3D consistency. Extensive experimental results demonstrate that our method outperforms existing state-of-the-art techniques in terms of lip movement accuracy, expression variation, and style feature expressiveness.
BibTeX
@article{ma2026esgaussianface,
  author = {Chuhang Ma and Shuai Tan and Ye Pan and Jiaolong Yang and Xin Tong},
  title = {{ESGaussianFace: Emotional and Stylized Audio-Driven Facial Animation via 3D Gaussian Splatting}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2026},
  doi = {10.1109/tvcg.2026.3651640},
}
Figure