Abstract
VideoReTalking is a state-of-the-art system designed for high-quality, audio-driven talking head video editing in unconstrained environments. By decoupling the generation process into three sequential stages—expression neutralization, audio-driven lip-syncing, and identity-aware enhancement—the framework produces realistic videos with accurate lip synchronization and editable emotional expressions. Notably, the system operates "in the wild" without requiring per-speaker training or specific reference footage.
1. Introduction
The goal of visual dubbing is to modify a video of a speaker so that their lip movements align perfectly with a new, arbitrary audio track. This technology has significant implications for film localization, personalized avatars, and digital video editing. However, existing methods typically face three major hurdles: identity dependence (requiring hours of specific speaker data), information leakage (where original lip movements interfere with the new audio), and overall visual quality (often resulting in blurry or uncanny facial features).
VideoReTalking addresses these challenges through a "divide-and-conquer" pipeline that prioritizes structural stability and identity preservation.
2. Methodology
The system decomposes the complex task of lip-syncing into three specialized networks:
2.1 Expression Neutralization (D-Net)
To prevent "information leakage" from the original video, the system first neutralizes the speaker's face. Using a 3D Morphable Model (3DMM), the D-Net extracts the speaker's pose and expression and replaces the original expression with a canonical neutral template. This provides a clean, stabilized structural reference for subsequent stages, ensuring the model does not have to "fight" the original mouth movements.
2.2 Lip-sync Generation (L-Net)
The L-Net takes the neutralized video and the target audio to synthesize new lip movements. This network utilizes Fast Fourier Convolutions (FFC) to capture global context and Adaptive Instance Normalization (AdaIN) to inject audio features into the visual stream. Because the input is already neutralized, the L-Net can focus entirely on accurate audio-to-visual mapping.
2.3 Identity-aware Enhancement (E-Net)
To achieve photo-realism, the final stage upscales the generated frames. Unlike generic face restorers that may subtly alter a person's appearance, the E-Net uses an identity-aware encoder. It leverages features from the original video to modulate the restoration process, ensuring that the final output maintains the specific identity and nuances of the speaker.
3. Datasets and Evaluation
The system was trained and evaluated using several prominent talking-head datasets:
- VoxCeleb: Utilized for training the expression reenactment (D-Net).
- LRS2 (Lip Reading Sentences 2): Used for training the lip-sync (L-Net) and enhancement (E-Net) components.
- HDTF (High-Resolution Deepfake): A high-resolution dataset used for evaluation to demonstrate effectiveness on 720p and 1080p content.
4. Key Insights
- The Importance of Neutralization: The authors identified that information leakage from the source video is a primary cause of artifacts. By forcing a neutral expression first, they eliminate the "ghosting" of original speech.
- Emotional Controllability: Because the system utilizes expression coefficients, the neutral template can be swapped for emotional templates (e.g., happy, angry). This allows the system to modify the speaker's emotion to match the tone of the new audio.
- Identity Preservation: High resolution alone is insufficient for realism; the E-Net architecture prioritizes identity consistency by feeding original facial features back into the upscaling process.
5. Results
VideoReTalking outperforms existing generic methods like Wav2Lip and PC-AVS in both quantitative metrics and qualitative user studies:
- Visual Quality: On the HDTF dataset, VideoReTalking achieved a Fréchet Inception Distance (FID) of 4.504, significantly better than Wav2Lip’s 5.632, indicating superior image realism.
- Lip-Sync Accuracy: The system achieved a Lip Sync Error - Confidence (LSE-C) score of 7.089 on LRS2, outperforming the baseline Wav2Lip (6.870).
- User Preference: In a study with 51 participants, VideoReTalking received a mean opinion score of 4.17/5 for visual quality, compared to 3.17/5 for Wav2Lip.
6. Limitations and Future Work
Despite its performance, the system has limitations. The dense warping in D-Net can occasionally cause subtle identity shifts. Additionally, extreme head poses can lead to inaccuracies in 3D reconstruction. Because enhancement is performed largely on a frame-by-frame basis, minor temporal jitter or "flashing" may occur. Future research aims to automate the selection of facial emotions based on the semantic context of the input audio.
Summary updated