Introduction
Automatic Video Dubbing (AVD) is the task of synthesizing human speech from a text script such that it is perfectly synchronized with a speaker's lip movements in a video. Traditional dubbing, often called Automated Dialogue Replacement (ADR), is a labor-intensive manual process where actors must precisely match their vocal performance to pre-recorded visual cues. From a computational perspective, AVD presents significant challenges: temporal synchronization, prosody control, and timbre matching. Standard text-to-speech (TTS) models are "one-to-many" and cannot inherently determine the precise timing required to match visual lip movements. Neural Dubber addresses these challenges by utilizing both the script and the video as primary inputs to bridge the gap between text and visual performance.
Methodology
Neural Dubber is a multi-modal, non-autoregressive TTS model designed to leverage visual information to guide speech synthesis. Its architecture introduces several key innovations to ensure both temporal alignment and speaker identity matching.
Text-Video Aligner
Unlike traditional TTS models that rely on duration predictors to estimate phoneme length, Neural Dubber introduces a Text-Video Aligner. This module employs a cross-modal attention mechanism to calculate the relationship between video frames (specifically the mouth regions) and phonemes. By treating the video frames as a "master clock," the model aligns phonemes to visual "visemes," ensuring that the synthesized speech speed and prosody are explicitly controlled by the lip movements.
Image-based Speaker Embedding (ISE)
To handle scenarios with multiple speakers, the ISE module predicts a speaker's voice timbre directly from a face image. Utilizing a pre-trained face recognition network, it extracts facial features and transforms them into an embedding. This allows the model to generate a plausible voice for a speaker even without prior audio recordings, effectively matching timbre to visual characteristics like age and gender.
Experiments and Results
The model was evaluated on the Chemistry Lecture (Chem) dataset (single speaker, ~9 hours) and the LRS2 dataset (multi-speaker, ~29 hours).
Synchronization and Quality
In Mean Opinion Score (MOS) tests for audio-visual synchronization, Neural Dubber achieved a score of 3.91, significantly higher than the 3.29 achieved by FastSpeech 2. Quantitative metrics using SyncNet (LSE-D and LSE-C) further validated that Neural Dubber's alignment is nearly indistinguishable from ground-truth recordings. The model also maintained high-fidelity audio quality, with a MOS of 3.74, demonstrating that visual constraints do not compromise naturalness.
Multi-speaker Performance
On the challenging LRS2 dataset, Neural Dubber outperformed modified baselines in audio quality (3.58 vs 3.15). This success highlights the effectiveness of the ISE module in capturing speaker identity from visual data alone, which is crucial for datasets with many speakers and limited audio per individual.
Conclusion
Neural Dubber represents a significant step forward in automatic video dubbing by treating video as a temporal guide for speech synthesis. While highly effective, the model currently depends on scripts that match the original video content and does not yet support translation dubbing. Future work aims to extend this capability to synthesize new lip movements that match translated text, further automating the global distribution of video content.
Summary updated