TL;DR
This paper presents a novel automatic dubbing system that goes beyond simple translation by ensuring the dubbed speech is naturally synchronized with the original video's timing and the speaker's vocal timbre. By integrating a pause-aware selection mechanism, LLM-based translation refinement, and voice cloning, the system achieves state-of-the-art lip-sync accuracy, outperforming previous methods.
What Problem Does This Paper Address?
Automatic dubbing aims to translate spoken content in a video into another language while maintaining temporal synchronization with the original speaker's lip movements. This "isochrony" is crucial for a natural viewing experience. Traditional approaches often struggle to balance translation quality with precise timing, especially when dealing with varying speech durations and the need to match vocal characteristics. Existing methods that try to incorporate timing constraints often fail to leverage powerful pre-trained models like large language models (LLMs) and machine translation (MT) models, which are trained on massive text corpora. Furthermore, creating datasets with detailed phoneme duration labels for training is challenging, limiting translation quality. This paper addresses the need for a more sophisticated dubbing system that can effectively utilize these advanced models while ensuring seamless temporal and vocal alignment.
Methods
The proposed system is a multi-component pipeline designed to tackle the challenges of automatic dubbing:
- Machine Translation (MT): A Transformer-based MT model, specifically a fine-tuned version of NLLB-1.3B on the CoVoST2 dataset, handles the initial translation from the source to the target language.
- LLM-based Post-Editor: A large language model (LLM) is employed to refine the translations. It's prompted to adjust the length of the translated text, offering "complex" or "simple" versions. This allows for generating translation candidates of varying lengths, which are crucial for optimizing lip-sync.
- Pause-Aware Dubbing Pipeline: This is a core innovation. It integrates a "pause selector" that identifies potential split points within the translated text. This selector uses sentence parsing (identifying prepositional phrases, verb phrases, and noun phrases) to ensure that natural linguistic units are not broken. It then selects the best split point that aligns with pauses in the original speech, minimizing the temporal mismatch.
- Text-to-Speech (TTS) and Voice Cloning (VC): A VITS-based TTS model is used for synthesizing the translated speech. Crucially, a voice cloning module based on OpenVoice is integrated to emulate the original speaker's vocal timbre, ensuring the dubbed speech sounds like the original speaker.
- Reranking: Multiple translation and dubbing candidates are generated. These candidates are then reranked using a combination of Lip-Sync Error Distance (LSE-D) and Character Error Rate (CER) to select the best overall result.
Datasets
- CoVoST2: This dataset was used for fine-tuning the NLLB-1.3B MT model for the German-English (DE-EN) translation task.
- IWSLT 2024: The system was submitted to the IWSLT 2024 automatic dubbing task, and evaluation was performed on DE-EN test sets (subset1 and subset2).
Key Insights and Ideas
- Disentangling MT and Dubbing Synchrony: The paper argues that a separate approach to MT and dubbing synchrony, rather than a unified model, can yield better results by leveraging powerful pre-trained MT and LLM models.
- Pause-Awareness is Crucial for Isochrony: Simply translating and synthesizing speech isn't enough. Identifying and aligning with natural pauses in the original speech is key to achieving seamless dubbing.
- LLMs for Length Control: LLMs, with their strong paraphrasing capabilities, are effective for generating translations of varying lengths, which is essential for optimizing lip-sync. The "complex" and "simple" prompts proved more stable than direct "longer" or "shorter" requests.
- Voice Cloning Enhances Perceptual Quality: Emulating the original speaker's vocal timbre significantly improves the naturalness and perceived quality of the dubbed speech.
- Reranking Balances Accuracy and Intelligibility: Combining LSE-D (for lip-sync) and CER (for speech intelligibility) in a reranking step provides a robust way to select the best dubbing candidate.
Key Results
The proposed pause-aware automatic dubbing system achieved impressive results on the DE-EN test sets:
- Lip-Sync Error Distance (LSE-D): The system achieved LSE-D scores of 10.75 on subset1 and 12.19 on subset2. These results are noted as being superior to the best system from the previous year.
- Character Error Rate (CER): The system achieved CERs of 4.62% on subset1 and 3.02% on subset2.
- MT Performance: The fine-tuned NLLB-1.3B model achieved strong MT performance with BLEU scores of 46.37 and 44.03 and Comet scores of 89.29 and 88.01 on subset1 and subset2, respectively.
- LLM Impact: While LLM post-editing slightly decreased BLEU scores, Comet scores remained similar, indicating effective paraphrasing without meaning alteration.
Limitations and Future Work
- Audio Super-Resolution Trade-off: While audio super-resolution was used to enhance audio quality, it was found to negatively impact LSE-D and CER, suggesting a trade-off between perceptual quality and synchronization accuracy.
- Expressiveness: The paper notes that for movie dubbing, matching the emotion of the original speech is critical. While they evaluated an expressive TTS model, its quality was inconsistent, especially for non-English languages.
- Future Work: The authors plan to conduct more systematic experiments on long-form, movie-like videos and aim to provide more expressive dubbed videos in the future. They also mention the challenges of the English-Chinese subtask, which involves more complex factors like speaker changes and background music, and suggest future work could involve diarization and source separation tools for such scenarios.
Summary updated