Abstract
Generating realistic talking face videos from a single image and arbitrary audio remains a significant challenge in multi-modal fusion. We present a real-time (60 FPS) framework utilizing Dynamic Convolution Kernels (DCKs). By predicting convolution weights directly from audio features, our model achieves precise lip synchronization and robust identity preservation across diverse head poses. Quantitative evaluations on the LRW benchmark demonstrate state-of-the-art performance in image quality and synchronization accuracy, outperforming existing methods in both visual fidelity and user preference.
Introduction
The synthesis of "talking head" videos—where a single portrait is animated by an arbitrary audio clip—is a cornerstone of digital human research. The primary difficulty lies in maintaining perfect synchronization between auditory phonemes and visual lip movements while preserving the speaker's unique identity and natural head motion.
Existing methodologies generally fall into two categories. Prior-based methods utilize intermediate representations like 2D facial landmarks or 3D face models. While these can be accurate, they are computationally intensive and often struggle with the "uncanny valley" effect. Conversely, direct fusion methods concatenate audio and image features. However, these often fail to produce sharp results when the face is not perfectly aligned, such as during head tilts or rotations. This paper introduces an end-to-end solution that is fast enough for real-time applications and robust to diverse head poses without requiring slow 3D reconstructions.
Methodology
The core innovation of this work is the Dynamic Convolution Kernel (DCK). In a standard Convolutional Neural Network (CNN), kernels are fixed after the training phase. In the proposed architecture, the kernels are dynamic, with their weights predicted in real-time based on the input audio.
As illustrated in Figure 1, the DCK approach differs fundamentally from simple feature concatenation or prior-based modeling. The system employs an "Adapted U-Net" where a pre-trained audio network (Wav2Lip) extracts features from the audio's Mel Spectrogram. These features are then processed by a Fully Connected (FC) network that generates the weights for the convolution kernels in the U-Net's bottleneck layers. Essentially, the audio "instructs" the visual network on how to filter the image, allowing for instantaneous adaptation to different phonemes.
The Dynamic Convolution Kernel Mechanism
By treating the network's parameters as dynamic variables—where the kernel $k$ is a function of the audio $h(Audio)$—the model effectively becomes a "meta-network." This allows it to approximate an infinite space of specific talking-head tasks.
As shown in Figure 4, while traditional layers use fixed kernels independent of the audio modality, the DCK layer integrates audio-driven kernels. This design provides several advantages:
- Translation Invariance: Because the architecture is fully convolutional, it is naturally robust to the face's position within the frame. The authors provide a mathematical proof (Theorem 1) showing that DCKs maintain lip-sync accuracy even as the subject moves their head.
- Blending for Realism: Instead of generating entire frames, the network outputs an attention mask and a motion image. These are blended with the original input frame, preserving background details and fine facial textures that are often lost in full-image generation.
Experimental Results
The model was trained on a mixed dataset of 550 real talking-face videos and evaluated against standard benchmarks like LRW (Lip Reading in the Wild). The proposed method demonstrates significant improvements over previous models such as Wav2Lip, ATVGnet, and X2Face across several key metrics:
- Efficiency: The system achieves 60 FPS for full-frame generation and up to 212 FPS for the face region alone, enabling true real-time performance.
- Visual Fidelity: The model achieved the highest scores in PSNR (31.98) and SSIM (0.81), indicating superior structural integrity and image clarity.
- Synchronization Accuracy: A low Landmark Distance (LMD) of 1.44 confirms high precision in lip-syncing.
- Human Perception: In a perceptual study, users preferred this method 80.2% of the time for visual quality and 73.0% for overall naturalness compared to competing state-of-the-art models.
Conclusion
The Dynamic Convolution Kernel strategy offers a powerful new paradigm for multi-modal synthesis. By making the network parameters themselves responsive to audio input, the system achieves a level of flexibility and real-time performance previously unattainable. While currently reliant on a pre-trained audio expert, future work may involve extending DCKs to more complex architectures like ResNets and exploring their application to extreme facial expressions beyond standard speech.
Summary updated