Neural Dubber: Dubbing for Silent Videos According to Scripts

Chenxu Hu, Qiao Tian, Tingle Li, Yuping Wang, Yuxuan Wang, Hang Zhao

Introduction

Automatic Video Dubbing (AVD) is the task of synthesizing human speech from a text script such that it is perfectly synchronized with a speaker's lip movements in a video. Traditional dubbing, often called Automated Dialogue Replacement (ADR), is a labor-intensive manual process where actors must precisely match their vocal performance to pre-recorded visual cues. From a computational perspective, AVD presents significant challenges: temporal synchronization, prosody control, and timbre matching. Standard text-to-speech (TTS) models are "one-to-many" and cannot inherently determine the precise timing required to match visual lip movements. Neural Dubber addresses these challenges by utilizing both the script and the video as primary inputs to bridge the gap between text and visual performance.

Methodology

Neural Dubber is a multi-modal, non-autoregressive TTS model designed to leverage visual information to guide speech synthesis. Its architecture introduces several key innovations to ensure both temporal alignment and speaker identity matching.

The architecture of Neural Dubber, including the Text-Video Aligner and the Image-based Speaker Embedding (ISE) module.

Text-Video Aligner

Unlike traditional TTS models that rely on duration predictors to estimate phoneme length, Neural Dubber introduces a Text-Video Aligner. This module employs a cross-modal attention mechanism to calculate the relationship between video frames (specifically the mouth regions) and phonemes. By treating the video frames as a "master clock," the model aligns phonemes to visual "visemes," ensuring that the synthesized speech speed and prosody are explicitly controlled by the lip movements.

Image-based Speaker Embedding (ISE)

To handle scenarios with multiple speakers, the ISE module predicts a speaker's voice timbre directly from a face image. Utilizing a pre-trained face recognition network, it extracts facial features and transforms them into an embedding. This allows the model to generate a plausible voice for a speaker even without prior audio recordings, effectively matching timbre to visual characteristics like age and gender.

Experiments and Results

The model was evaluated on the Chemistry Lecture (Chem) dataset (single speaker, ~9 hours) and the LRS2 dataset (multi-speaker, ~29 hours).

Comparison of mel-spectrograms generated by Ground Truth (GT), Neural Dubber (ND), FastSpeech 2 (FS2), and Video-based Tacotron (VT).

Synchronization and Quality

In Mean Opinion Score (MOS) tests for audio-visual synchronization, Neural Dubber achieved a score of 3.91, significantly higher than the 3.29 achieved by FastSpeech 2. Quantitative metrics using SyncNet (LSE-D and LSE-C) further validated that Neural Dubber's alignment is nearly indistinguishable from ground-truth recordings. The model also maintained high-fidelity audio quality, with a MOS of 3.74, demonstrating that visual constraints do not compromise naturalness.

Multi-speaker Performance

On the challenging LRS2 dataset, Neural Dubber outperformed modified baselines in audio quality (3.58 vs 3.15). This success highlights the effectiveness of the ISE module in capturing speaker identity from visual data alone, which is crucial for datasets with many speakers and limited audio per individual.

Conclusion

Neural Dubber represents a significant step forward in automatic video dubbing by treating video as a temporal guide for speech synthesis. While highly effective, the model currently depends on scripts that match the original video content and does not yet support translation dubbing. Future work aims to extend this capability to synthesize new lip movements that match translated text, further automating the global distribution of video content.

Summary updated

BibTeX
@inproceedings{hu2021neural,
  author = {Chenxu Hu and Qiao Tian and Tingle Li and Yuping Wang and Yuxuan Wang and Hang Zhao},
  title = {{Neural Dubber: Dubbing for Silent Videos According to Scripts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=11G1RZzuQy},
}
Figure