TL;DR
This paper introduces a novel prototype for simultaneous dubbing of live video streams, translating and replacing original speech with synthesized speech in real-time. The system achieves a low average latency of 11.90 seconds, offering a smoother cross-language viewing experience than traditional subtitles.
What Problem Does This Paper Address?
Live video streaming has become a ubiquitous communication tool, but language barriers remain a significant hurdle for global engagement. The current standard solution, automatic simultaneous interpretation via subtitles, often degrades the viewing experience by requiring users to read text at the bottom of the screen. This paper addresses the challenge of providing a more seamless cross-language experience by developing a prototype that performs automatic simultaneous dubbing, effectively replacing the original audio with a translated and synthesized version in real-time.
Methods
The prototype achieves simultaneous dubbing through a three-step pipeline:
- Sentence Segmentation: The source video stream is segmented into individual sentences using automatic speech recognition (ASR) and sentence segmentation techniques. The authors leverage Transformer-based acoustic models and seq2seq criteria for ASR, and a Transformer backbone for sentence segmentation.
- Translated Speech Generation: For each segmented sentence, a translated speech waveform is generated. This involves machine translation (MT) using a Transformer model (implemented in OpenNMT) and text-to-speech (TTS) synthesis. The TTS component modifies the VITS implementation to generate speech waveforms from speaker embeddings, aiming to match the original voice characteristics.
- Adaptive Playback: The generated translated speech waveforms are played back along with the video images using an adaptive playback method. This method dynamically adjusts the playback speed to manage the accumulated data in the playback buffer, balancing latency and smoothness. The core idea is to speed up playback when the buffer is low and maintain normal speed otherwise, controlled by parameters like playback acceleration and buffer size threshold.
Datasets
The system was tested on a collection of 90 public videos, comprising approximately 21 hours and 45 minutes of Japanese interviews, speeches, presentations, and lectures. No new datasets were introduced; the focus was on evaluating the prototype's performance on existing video content.
Key Insights and Ideas
- Dubbing Over Subtitles: The core insight is that replacing audio with dubbed speech offers a more immersive and less distracting cross-language experience compared to reading subtitles, especially for live content.
- Adaptive Playback for Real-time Constraints: The challenge of irregular output timing from MT and TTS is effectively addressed by an adaptive playback mechanism. This allows for continuous video streaming while managing the inherent delays in speech processing.
- Trade-offs in Latency and Smoothness: The paper highlights the inherent trade-off between latency and smoothness. The system offers different modes (Fast, Balance, Quality) to allow users to choose their preferred balance based on application needs.
- Language Pair Differences: The authors note that for languages with significant structural differences (like Japanese and English), forcing translations to match source sentence length can degrade quality. This suggests that a flexible approach to duration matching is crucial.
Key Results
The prototype achieved a low average latency of 11.90 seconds in the "Fast" mode, meeting a smoothness criterion. This is considered fast given the average sentence duration of 10.76 seconds and maximum delay of 15.59 seconds. The system also demonstrated good duration matching, with the "Fit (%)" metric (percentage of translated waveforms fitting original durations) reaching up to 91.50% in the "Quality" mode. The "Quality" mode, by employing multiple translation models, showed improved Fit and D. Fit percentages while decreasing D. Ex. (duration exceedance).
Limitations and Future Work
- Limitations: The paper acknowledges that for the Japanese-English language pair, forcing translations to match source sentence length can significantly degrade translation quality. The current system does not adjust speech rate to avoid sounding unnatural.
- Future Work: The authors plan to incorporate optical character recognition (OCR) to translate on-screen text (like lecture slides) into the target language, providing a more complete viewing experience. They also mention the potential for extending the system to other language pairs.
Summary updated