FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization

Shuai Tan, Bin Ji, Ye Pan

Abstract

Generating emotional talking faces is a practical yet challenging endeavor. To create a lifelike avatar, we draw upon two critical insights from a human perspective: 1) The connection between audio and the non-deterministic facial dynamics, encompassing expressions, blinks, poses, should exhibit synchronous and one-to-many mapping. 2) Vibrant expressions are often accompanied by emotion-aware high-definition (HD) textures and finely detailed teeth. However, both aspects are frequently overlooked by existing methods. To this end, this paper proposes using normalizing Flow and Vector-Quantization modeling to produce emotional talking faces that satisfy both insights concurrently (FlowVQTalker). Specifically, we develop a flow-based coefficient generator that encodes the dynamics of facial emotion into a multi-emotion-class latent space represented as a mixture distribution. The generation process commences with random sampling from the modeled distribution, guided by the accompanying audio, enabling both lip-synchronization and the uncertain nonverbal facial cues generation. Furthermore, our designed vector-quantization image generator treats the creation of expressive facial images as a code query task, utilizing a learned codebook to provide rich, high-quality textures that enhance the emotional perception of the results. Extensive experiments are conducted to showcase the effectiveness of our approach.

Introduction

Generating a talking head from a single image and an audio clip is a staple of modern AI research, but achieving true human-like realism remains a significant hurdle. Most existing models suffer from two "uncanny valley" symptoms: they are overly deterministic, producing robotic, repetitive movements, and they lack the fine-grained textures—such as forehead crinkles or clear dental details—that signal genuine emotion.

System overview of FlowVQTalker, highlighting diverse facial dynamics and high-definition textures.

Researchers from Shanghai Jiao Tong University address these gaps with FlowVQTalker, a framework that combines the probabilistic power of Normalizing Flows with the high-fidelity detail of Vector Quantization. By modeling the "one-to-many" relationship between audio and facial expressions, FlowVQTalker produces realistic, emotional talking head videos with high-definition textures that are often lost in traditional methods.

Methodology

FlowVQTalker decomposes the generation task into two specialized modules, bridged by 3D Morphable Model (3DMM) coefficients.

Flow-based Coefficient Generator (FCG)

To capture the non-deterministic nature of human expression, the FCG utilizes Normalizing Flows. The sub-modules ExpFlow and PoseFlow map audio and emotion labels into a latent space. Unlike standard models that use Gaussian distributions, FlowVQTalker employs a Student’s t Mixture Model (SMM). The "fat tails" of the Student’s t-distribution better handle outliers in small emotional datasets, preventing overfitting and allowing the system to stochastically sample plausible facial dynamics, such as natural blinks, for the same audio input.

Vector-Quantized Image Generator (VQIG)

To resolve the "texture gap," the authors move away from standard pixel-wise reconstruction toward a code query task. Inspired by VQ-GAN, the VQIG utilizes a discrete codebook trained on the FFHQ dataset to store high-quality visual "atoms" of facial features. The model warps the source image based on predicted motion and uses a Multi-Head Cross-Attention (MHCA) mechanism and Adaptive Instance Normalization (AdaIN) to fuse identity from the source image with high-fidelity textures retrieved from the codebook. This ensures that details like teeth and emotion-specific wrinkles are rendered with clarity.

Evaluation and Results

The model was evaluated on high-standard benchmarks including MEAD (emotional audio-visual data) and HDTF (high-resolution talking faces).

Qualitative comparison of FlowVQTalker against state-of-the-art methods, demonstrating superior texture and emotional expression.

Quantitative and Qualitative Performance

FlowVQTalker demonstrates a significant leap over state-of-the-art models such as Wav2Lip, EAMM, and SadTalker.

  • Texture Sharpness: It achieved the highest scores in Cumulative Probability Blur Detection (CPBD), indicating superior image clarity.
  • Emotional Accuracy: The model led in Emotion Accuracy (Acc_emo), effectively conveying intended emotions more reliably than competitors.
  • Visual Detail: Qualitative comparisons show that FlowVQTalker is uniquely capable of rendering distinct teeth and subtle skin deformations associated with specific emotions, which other models often blur.

A user study with 20 participants further confirmed these findings, ranking FlowVQTalker as the most realistic in terms of both lip-sync and overall image quality.

Conclusion and Future Work

FlowVQTalker successfully bridges the gap between deterministic motion and high-fidelity emotional expression. While the model sets a new standard, it relies on 3DMM coefficients, which have inherent limits in representing extreme deformations or complex hair movements. Future work may explore moving beyond 3DMM constraints and optimizing the transformer-based VQ generator for real-time applications on consumer hardware.

Summary updated

Abstract

Generating emotional talking faces is a practical yet challenging endeavor. To create a lifelike avatar, we draw upon two critical insights from a human perspective: 1) The connection between audio and the non-deterministic facial dynamics, encompassing expressions, blinks, poses, should exhibit synchronous and one-to-many mapping. 2) Vibrant expressions are often accompanied by emotion-aware high-definition (HD) textures and finely detailed teeth. However, both aspects are frequently overlooked by existing methods. To this end, this paper proposes using normalizing Flow and Vector-Quantization modeling to produce emotional talking faces that satisfy both insights concurrently (FlowVQTalker). Specifically, we develop a flow-based coefficient generator that encodes the dynamics of facial emotion into a multi-emotion-class latent space represented as a mixture distribution. The generation process commences with random sampling from the modeled distribution, guided by the accompanying audio, enabling both lip-synchronization and the uncertain nonverbal facial cues generation. Furthermore, our designed vector-quantization image generator treats the creation of expressive facial images as a code query task, utilizing a learned codebook to provide rich, high-quality textures that enhance the emotional perception of the results. Extensive experiments are conducted to showcase the effectiveness of our approach.
BibTeX
@inproceedings{tan2024flowvqtalker,
  author = {Shuai Tan and Bin Ji and Ye Pan},
  title = {{FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=L3MwMLC9Xf},
}
Figure