TL;DR
This paper tackles the challenge of real-time speech-to-speech translation (S2ST) by significantly reducing latency. They achieve this by having the speech translator generate "pseudo lookahead" for the speech synthesizer, minimizing initial delays, and then employing a duration-scaling technique to further cut down on overall latency without sacrificing translation quality.
What Problem Does This Paper Address?
Simultaneous speech-to-speech translation (S2ST) aims to convert spoken input in one language to spoken output in another, in real-time. A major hurdle is the cumulative delay introduced by the different components of the translation pipeline, particularly the speech translation (ST) module and the incremental text-to-speech (iTTS) module. While iTTS models have improved, they often rely on "lookahead" – information about future text – to produce high-quality speech. However, this lookahead itself introduces an initial delay. This paper addresses the problem of minimizing both the initial delay of the iTTS module and the overall latency, considering that the duration of synthesized speech also significantly impacts how quickly a translation can be fully delivered to the user.
Methods
The authors propose two main strategies to reduce latency:
- Input-Speech-Guided Pseudo Lookahead: Instead of relying solely on language models to predict future text for the iTTS system (which can be inaccurate), they leverage the upstream speech translator. The ST system, which already has access to the source speech, generates "pseudo lookahead" – predicted future text tokens – that are more contextually relevant. This pseudo lookahead helps the iTTS module start synthesizing speech earlier and more accurately, mitigating the initial delay caused by traditional lookahead mechanisms.
- Speech-Duration-Aware Latency Metric and Duration Scaling: The paper introduces a latency metric that accounts for both the start time and the end time of synthesized speech, recognizing that long speech durations can lead to significant overall latency, even if the system starts quickly. Based on this, they propose a duration-scaling approach. Using non-autoregressive TTS models (like FastSpeech 2), which allow for explicit control over speech duration, they can speed up the synthesized speech by scaling down its duration. This is particularly useful when the input text arrives at the iTTS module at a faster rate than the original ST system's output rate.
Additionally, they improve their iTTS model by augmenting the training data with prefixes and by explicitly signaling full versus partial input sentences to the model, leading to better prosody prediction.
Datasets
- ST Data:
- Fisher Spanish-English (es→en)
- MuST-C v1.0 (en→es)
- TTS Data:
- LJSpeech (English)
- CSS10 Spanish (es)
The paper also mentions using the Montreal Forced Aligner (MFA) for phoneme duration extraction.
Key Insights and Ideas
- Leveraging Source Speech for Lookahead: The most significant insight is that the source speech itself, processed by the ST system, provides a richer context for generating pseudo lookahead for the TTS system than standalone language models. This allows for more accurate predictions and reduces the need for actual future text, thus cutting initial latency.
- Duration is King for Latency: The paper strongly emphasizes that simply reducing the start time of synthesized speech isn't enough. The duration of that speech is a critical factor in the overall delivery time, especially in continuous translation scenarios.
- Pseudo Lookahead Can Match Ground Truth: For the first time, the authors demonstrate that pseudo lookahead generated by the ST system can achieve output speech quality comparable to using actual future text (ground-truth lookahead), while avoiding the associated delay.
- Duration Scaling is Effective: Speeding up synthesized speech by scaling its duration is a viable strategy to counteract latency introduced by faster incoming text, without significantly degrading quality.
Key Results
- Latency Reduction: The proposed approaches consistently reduce latency by 0.2-0.5 seconds.
- Quality Preservation: This latency reduction is achieved without sacrificing speech translation quality.
- Pseudo Lookahead Performance: The ST-generated pseudo lookahead matches the quality of using actual lookahead (e.g., "Lookahead 1 word") in S2ST tasks.
- Duration Scaling Impact: In experiments, moderately speeding up output speech (scaling durations by 0.95 or 0.9) counteracted latency induced by faster incoming text, reducing latency by approximately 0.7 seconds without noticeable degradation in MOS or CER.
- Improved iTTS Model: Prefix augmentation and EOS token signaling improved the iTTS model's performance across various decoding strategies.
Limitations and Future Work
- Accuracy of Pseudo Lookahead: While the ST-generated pseudo lookahead is shown to be beneficial, the paper notes that inaccurate pseudo lookahead (e.g., random tokens) can be detrimental to synthesis quality. Future work could focus on further improving the accuracy and robustness of this pseudo lookahead generation.
- Naturalness vs. Speed: The paper acknowledges that while duration scaling can reduce latency, there's a trade-off with naturalness and intelligibility. The current work shows minimal impact with moderate scaling, but pushing this further might require more sophisticated control mechanisms.
- Real-time Deployment: While the paper focuses on latency reduction, the practical deployment of these techniques in a full real-time S2ST system would involve further engineering considerations.
Summary updated