End-to-End Multilingual Automatic Dubbing via Duration-based Translation with Large Language Models

Hyun-Sik Won, DongJin Jeong, Hyunkyu Choi, Jinwon KimEMNLP 2025

TL;DR

This paper introduces a novel end-to-end automatic dubbing system that leverages Large Language Models (LLMs) to achieve natural synchronization between translated speech and original video content. By dynamically adjusting translation length based on source speech duration, their "Duration-based Translation" (DT) method significantly improves temporal alignment without sacrificing translation quality, offering a more practical solution for multilingual dubbing.

What Problem Does This Paper Address?

Automatic dubbing (AD) aims to replace a video's original audio with a translated version while maintaining temporal synchronization, a crucial aspect known as isochrony. This means the dubbed speech should align with the speaker's mouth movements and pauses in the original video. A major challenge is that different languages have varying speech durations for equivalent content. Traditional AD systems often use a pipeline of Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). However, NMT typically prioritizes lexical accuracy over temporal constraints, leading to translations that don't match the original speech's timing. Existing solutions like adjusting TTS speaking rates can sound unnatural, while methods that match character or word counts don't guarantee similar speech durations due to language-specific speaking rates. Other approaches, while improving isochrony, often add complexity and require language-specific tuning, hindering scalability.

Methods

The core innovation is an end-to-end automatic dubbing framework that integrates translation and temporal control using LLMs. The key component is Duration-based Translation (DT). This method dynamically predicts the optimal number of phonemes for a translated segment based on the source speech's duration and the target TTS system's speaking rate.

The pipeline involves:

  1. Speech-to-Text (STT): Using Whisper to transcribe audio and extract precise word-level timing annotations.
  2. Neural Machine Translation (NMT) with Duration-based Translation (DT): This is the novel part. Instead of just translating, the LLM iteratively refines the translation. It first generates a reference translation and then, based on the predicted optimal phoneme count (derived from source duration and TTS rate), it either shortens or lengthens phrases. This is achieved by instructing the LLM to remove extraneous phrases or add modifiers. Multiple candidates are generated and the best one, balancing meaning preservation and phoneme count, is selected.
  3. Pause Integration: To further enhance naturalness, the system identifies pauses in the source speech and uses the LLM to determine appropriate pause placements in the translated text, considering linguistic structure.
  4. Text-to-Speech (TTS): Using ElevenLabs for synthesis, with quality scores from UTMOS to select appropriate voice samples and produce naturally synchronized dubbed audio.

Datasets

The paper uses the Multilingual Interpretation and Translation Reading-style Dataset provided by AI Hub. This dataset contains sentence triplets in Korean, English, and Spanish, designed to have natural variations in utterance durations due to linguistic differences. For experiments, they sampled 100 triplets per language, treating each language as a target and the other two as sources, enabling evaluation across multiple language pairs.

Key Insights and Ideas

  • Duration, not just length, is key: Simply matching character or word counts (isometric translation) is insufficient. The paper highlights that predicting phoneme counts based on actual speech duration is crucial for accurate temporal alignment.
  • LLMs as flexible translators: LLMs are powerful enough to not only translate but also to iteratively refine translations based on complex, non-lexical constraints like duration and phoneme count.
  • End-to-end is achievable: By integrating duration prediction and LLM-based iterative refinement directly into the translation module, the system avoids the need for separate, complex post-processing steps for timing.
  • Multilingual without tuning: The proposed DT approach is language-agnostic, making it highly practical for real-world multilingual dubbing scenarios.

Key Results

  • Improved Speech Overlap: The DT approach achieved the highest speech overlap scores across most translation directions, indicating significantly tighter temporal alignment compared to unconstrained translation (GPT-4o) and phoneme-count matching (PT). On average, DT improved speech overlap by approximately 16.15% over GPT-4o.
  • Competitive Translation Quality: While GPT-4o achieved slightly higher COMET scores (indicating translation quality), the DT approach maintained competitive quality while drastically improving temporal alignment.
  • DT vs. PT: DT showed a relative improvement of approximately 3.75% in speech overlap compared to PT, demonstrating the added benefit of considering source speech duration over just matching phoneme counts.
  • Human Evaluation: In a subjective evaluation (Mean Opinion Score - MOS), the proposed DT system scored higher (6.57) than a proprietary system (5.83), suggesting that its approach leads to more natural and coherent dubbed speech.

Limitations and Future Work

  • TTS Dependency: The system's performance is dependent on the specific speaking rate and phoneme-generation characteristics of the underlying TTS system. Changes in the TTS engine could directly impact temporal alignment quality, potentially requiring recalibration.
  • Future Work: The authors plan to investigate methods to reduce this dependency and improve robustness across different speech synthesis systems.

Summary updated

BibTeX
@inproceedings{won2025end,
  author = {Hyun-Sik Won and DongJin Jeong and Hyunkyu Choi and Jinwon Kim},
  title = {{End-to-End Multilingual Automatic Dubbing via Duration-based Translation with Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure