Incremental Speech Synthesis For Speech-To-Speech Translation

Danni Liu, Changhan Wang, Hongyu Gong, Xutai Ma, Yun Tang, Juan Miguel Pino

Introduction

Real-time speech-to-speech translation (S2ST) aims to provide instantaneous communication across language barriers. However, the user experience is often marred by significant latency, leading to unnatural pauses and disjointed conversations. This latency is largely driven by the incremental text-to-speech (iTTS) component, which must wait for the preceding speech-to-text (ST) module to produce enough text to generate natural-sounding audio. This paper introduces two innovative strategies to minimize this delay: Input-Speech-Guided Pseudo Lookahead and Duration Scaling. By predicting upcoming words and slightly accelerating audio playback, the authors achieve a reduction in latency of up to 0.5 seconds without compromising translation accuracy or speech naturalness.

The Latency Bottleneck in S2ST

In a simultaneous S2ST pipeline, the system processes audio in chunks. The iTTS module faces a unique challenge: to produce high-quality prosody (the rhythm and intonation of speech), it needs to know what words are coming next. For instance, the pitch of a word often changes depending on whether it is in the middle or at the end of a sentence. Waiting for these "lookahead" words from the ST module creates an inherent initial delay.

Furthermore, S2ST systems operate under a "non-overlapping constraint," meaning the system cannot begin playing the audio for a newly translated word until the current word's audio has finished playing. If the synthesized speech is slower than the rate at which the ST module produces text, a "traffic jam" occurs. Translated words begin to pile up in a buffer, and the overall system latency grows cumulatively as the sentence progresses.

Proposed Methodologies

The authors propose a multi-pronged approach to address latency at different stages of the pipeline.

1. Input-Speech-Guided Pseudo Lookahead

Traditional iTTS systems wait for the ST module to finalize the next word before beginning synthesis. The authors propose a "pseudo lookahead" mechanism where the ST module predicts the next token based on the current source audio input. Unlike generic language models (such as GPT-2) that guess the next word based solely on text history, this method is conditioned on the actual acoustic signal of the speaker. This grounding makes the predictions significantly more accurate, allowing the TTS model to begin synthesis earlier with high confidence in the future context.

Comparison of lookahead strategies for incremental speech synthesis, illustrating how pseudo lookahead leverages source speech for better prediction.

2. Duration-Aware Latency and Scaling

To combat the accumulation of latency during playback, the authors employ a non-autoregressive TTS model, specifically FastSpeech 2. This architecture allows for explicit control over the duration of each synthesized phoneme. By applying a scaling factor (e.g., 0.9x), the system can slightly speed up the output speech. This subtle acceleration helps clear the audio buffer faster than the ST module populates it, effectively preventing the "traffic jam" effect.

3. Prefix-Augmented Training

Synthesizing partial sentences is inherently difficult because the model doesn't know if a segment is a complete thought. To address this, the authors augmented their training data with sentence prefixes. They trained the model to recognize the end-of-sentence (EOS) token as a signal to apply final-sounding prosody, such as dropping intonation or lengthening the final syllable, while maintaining "mid-sentence" prosody for other segments.

Experimental Setup and Datasets

The proposed methods were validated using standard datasets for both translation and synthesis:

  • Speech-to-Text (ST): Fisher (Spanish-to-English) and MuST-C v1.0 (English-to-Spanish).
  • Text-to-Speech (TTS): LJSpeech (English) and CSS10 (Spanish).

Key Insights

One of the most significant findings was that attempting to eliminate lookahead entirely ("zero lookahead") actually increases total latency. Without context, the TTS model tends to stretch out word durations, which exacerbates the playback backlog.

Additionally, the study highlighted the importance of prediction accuracy. Using a text-only language model for lookahead often resulted in incorrect predictions that degraded translation quality. In contrast, the ST-guided pseudo lookahead was so accurate that it performed nearly as well as having the actual ground-truth future text.

Results

The integration of these techniques yielded substantial improvements in system performance:

  • Latency Reduction: The ST-guided pseudo lookahead strategy reduced latency by 0.2 to 0.5 seconds compared to standard lookahead methods.
  • Quality Preservation: These gains were achieved without a significant drop in translation quality, maintaining BLEU scores consistent with ground-truth lookahead.
  • Duration Scaling Impact: Scaling the output duration by 0.90x provided an additional latency reduction of approximately 0.7 seconds. Crucially, human evaluators found the slightly faster speech to be natural, with Mean Opinion Scores (MOS) remaining stable.
Impact of duration scaling on latency and Mean Opinion Score (MOS), demonstrating the trade-off between speed and naturalness.

Limitations and Future Directions

While the results are promising, the study is based on a cascaded pipeline architecture. As the field moves toward direct end-to-end S2ST models, future work will need to adapt these duration-control and lookahead strategies to those unified architectures. Furthermore, while TTS quality was evaluated via MOS, a holistic human evaluation of the entire translation experience—including the impact of reduced latency on conversational flow—remains an area for future exploration.

Summary updated

BibTeX
@inproceedings{liu2022incremental,
  author = {Danni Liu and Changhan Wang and Hongyu Gong and Xutai Ma and Yun Tang and Juan Miguel Pino},
  title = {{Incremental Speech Synthesis For Speech-To-Speech Translation}},
  booktitle = {Interspeech},
  year = {2022},
  doi = {10.21437/interspeech.2022-10568},
}
Figure