Abstract
This paper reports our solution for the ACM Multimedia ViCo 2022 Conversational Head Generation Challenge, which aims to generate vivid face-to-face conversation videos based on audio and reference images. Our solution focuses on training a generalized audio-to-head driver using regularization and assembling a high-visual quality renderer. We carefully tweak the audio-to-behavior model and post-process the generated video using our foreground-background fusion module. We achieved first place in the listening head generation track and second place in the talking head generation track on the official leaderboard.
Introduction
Generating realistic conversational head videos—where a digital persona talks or listens with natural responsiveness—is a multi-layered challenge in computer vision. Beyond simple lip-syncing, the task requires synthesizing natural head movements and expressions that align with the emotional and rhythmic nuances of the input audio.
Existing methodologies often encounter two primary hurdles. First, models frequently overfit on limited datasets, leading to jittery or unnatural movements when presented with novel audio. Second, the "uncanny valley" effect is often triggered by rendering artifacts, such as background warping or distortion around the head's edges during movement. This research presents a generalized system designed to produce vivid conversational videos from a single reference image and an audio stream, addressing these issues through a two-stage pipeline consisting of a regularized driver and an enhanced renderer.
Methodology
1. The Sequential Driver (Audio to 3DMM)
Rather than attempting to generate pixels directly from audio—a process prone to noise—the system first predicts 3D Morphable Model (3DMM) parameters, including expression, rotation, and translation.
- Architecture: The driver utilizes a 4-layer Long Short-Term Memory (LSTM) network to process audio features, including MFCCs, energy, and zero-crossing rates.
- Regularization and Residual Learning: To combat overfitting, the authors implemented Residual Learning, where the model predicts the change from the initial frame rather than absolute values. This is supplemented by Dropout and Batch Normalization.
- Temporal Continuity: A "head motion constraint loss" was introduced to ensure smooth transitions between frames, effectively eliminating the "bobblehead" effect often seen in automated generation.
2. The Enhanced Renderer (3DMM + Image to Video)
The rendering stage utilizes PIRenderer as a foundation, augmented by two critical enhancements to ensure visual stability:
- Foreground-Background Fusion: Using a pre-trained U2Net segmentation model, the system separates the subject from the background. By keeping the background static while the head moves, the model avoids the "melting background" distortion typical of neural rendering.
- Image Boundary Inpainting: To handle the "holes" or empty spaces that appear at frame edges during head rotation, the authors employed a border padding technique during feature warping to fill gaps seamlessly.
Experimental Results
Dataset and Training
The model was trained and validated using the ViCo dataset, which comprises 483 video clips (~95 minutes) featuring 76 listeners and 67 speakers. This dataset is uniquely suited for conversational tasks as it includes both active talking and responsive listening scenarios.
Performance and Ablation
The system's efficacy was confirmed on the official ViCo challenge leaderboard:
- Listening Head Track: 1st Place (FID: 21.350, ExpFD: 0.116).
- Talking Head Track: 2nd Place (FID: 24.678).
Ablation studies highlighted the necessity of the chosen architectural constraints. Specifically, the inclusion of residual learning improved the Landmark Distance (LMD) from 65.18 to 43.02, and increased PSNR by 1.46dB. Furthermore, the authors found that "cross-model ensembles" (averaging predictions from models across different training sessions) provided superior Fréchet Inception Distance (FID) scores compared to standard self-ensembles.
Discussion and Future Work
A key insight from this research is that human perception is highly sensitive to background stability; treating the background as a static layer significantly reduces the perceived artificiality of the video. However, limitations remain. The current model can be "conservative" with lip shapes, occasionally failing to distinguish between subtle syllables. Additionally, maintaining the specific identity of the reference image during extreme movements remains a challenge. Future iterations may focus on fine-tuning the renderer for specific character identities and improving the granularity of the audio-to-expression mapping.
Summary updated