The HW-TSC’s Simultaneous Speech-to-Speech Translation System for IWSLT 2023 Evaluation

Hengchao Shang, Zhiqiang Rao, Zongyao Li, Zhanglin Wu, Jiaxin Guo, Minghan Wang, Daimeng Wei, Shaojun Li, Zhengzhe Yu, Xiaoyu Chen, Lizhi Lei, Hao YangACL 2023

TL;DR

This paper presents HW-TSC's submission to the IWSLT 2023 Simultaneous Speech-to-Speech Translation (SimulS2S) task. They tackle the challenge of latency and audio quality in cascaded SimulS2S systems by introducing an "onlinization" technique and "deblanking" strategies to improve TTS output.

What Problem Does This Paper Address?

Simultaneous Speech-to-Speech Translation (SimulS2S) aims to translate spoken language into another spoken language in real-time, mimicking human interpreters. Traditional cascaded approaches break this down into three stages: Speech-to-Text (ASR), Text-to-Text Translation (MT), and Text-to-Speech (TTS). While effective, these cascaded systems face significant challenges, particularly in achieving low latency while maintaining high translation quality. A key issue is the "onlinization" of offline models, which are trained on complete utterances and struggle with the incremental nature of real-time translation. Furthermore, the TTS module can produce unnatural pauses or blank sounds, disrupting the flow of the translated speech. This paper addresses these issues by proposing methods to make cascaded systems more efficient and produce higher-quality audio output.

Methods

The HW-TSC system is a cascaded approach comprising three core components:

  • ASR Module: They utilize the U2 framework, which supports both streaming and non-streaming ASR. A key feature is its dynamic chunk training and decoding with a CTC decoder, allowing for better handling of streaming input. For decoding, they employ an "attention_rescoring" strategy, using the attention decoder to re-score CTC outputs via prefix beam search.
  • MT Module: The Transformer architecture is used for machine translation. To enhance its performance, they employ several strategies:
  • Multilingual Training: Training a single model for multiple language pairs (English-German, English-Chinese, English-Japanese).
  • Synthetic Data Generation: Using beam search decoding to generate synthetic data for training.
  • Domain Adaptation: Fine-tuning the MT model by replacing source-side text with data from a well-trained ASR model to reduce domain gaps.
  • TTS Module: The VITS model, a state-of-the-art text-to-speech system, is used for generating audio.
  • Onlinization Technique: To adapt offline models for real-time use, they implement an incremental decoding strategy. This involves dividing input utterances into fixed-size chunks and processing them sequentially. The system maintains a buffered decoder state to allow attention to span across chunks, improving context awareness.
  • Stable Hypothesis Detection: Inspired by prior work, they adapt strategies like "Hold-n" and "LA-n" to detect stable hypotheses in cascaded systems, aiming to minimize errors from incomplete input. This involves identifying common prefixes across hypotheses generated from different chunks.
  • Deblanking Strategies: To address issues with TTS output, specifically unusual waveforms and blank sounds, they introduce two strategies:
  • Unknown Filtering: Removing tokens not present in the vocabulary and converting Arabic numerals to textual numerals for Chinese.
  • Context-Aware Pause Detection: Analyzing the text to determine if a pause is indicated. If not, they eliminate prolonged silences in the generated audio, while reserving a minimum of 160 frames for speech coherence.

Datasets

The paper mentions the following datasets:

  • ASR Training: LibriSpeech V12, MuST-C V2, TEDLIUM V3, and CoVoST V2.
  • MT Training: Official parallel corpora, with a focus on data similar to the MuST-C domain.
  • TTS Inference: Pre-trained VITS models from Hugging Face for EN-JA/ZH, and Espnet for EN-DE.

Key Insights and Ideas

  • Cascaded Systems Can Be Competitive: Despite the rise of end-to-end models, this work demonstrates that well-tuned cascaded systems can still achieve strong results in SimulS2S tasks.
  • Onlinization is Crucial for Latency: Adapting offline models to streaming scenarios through incremental decoding and stable hypothesis detection is key to reducing latency without sacrificing too much quality.
  • Deblanking Improves Audio Naturalness: The proposed "deblanking" strategies are essential for producing more natural and less jarring TTS output, which is critical for user experience in SimulS2S. The context-aware pause detection, in particular, seems to be a clever way to avoid unnatural silences.
  • Multilingual and Data Augmentation Techniques Pay Off: The use of multilingual training and synthetic data generation for MT highlights the importance of leveraging available data effectively.

Key Results

The paper presents results in Table 1 and Table 2.

  • Table 1 (Final Systems Results):
    • For EN-JA, their system achieved a BLEU score of 14.53, with StartOffset and EndOffset metrics of 1.59 and 2.96 respectively.
  • For EN-ZH, they achieved a BLEU score of 20.19, with StartOffset and EndOffset metrics of 1.77 and 2.98 respectively.
  • The paper notes that TTS transcription results in Japanese had the smallest gap compared to the S2T system (around 3 BLEU difference), while German showed the worst results.
  • Table 2 (Ablation Study on Deblanking Strategies):
    • For EN-DE, the baseline BLEU was 10.45. "Context-aware wait" yielded 10.32, and "Unknown Filtering" yielded 10.27.
  • For EN-JA, the baseline was 14.53. "Context-aware wait" achieved 13.37, and "Unknown Filtering" achieved 13.08.
  • For EN-ZH, the baseline was 20.19. "Context-aware wait" achieved 18.64, and "Unknown Filtering" achieved 16.73.
  • These results strongly suggest that the proposed deblanking strategies are effective in reducing the gap between offline and streaming TTS performance.

Limitations and Future Work

  • Limitations:
    • The paper notes that the TTS system generated the worst results among all evaluated systems for the German direction, indicating a need for further investigation into the underlying reasons and potential improvements.
  • While the deblanking strategies improved TTS, there's still a gap between offline and streaming TTS performance, as seen in the ablation study.
  • Future Work:
    • The authors express a desire to delve further into end-to-end systems in the future.
  • Further research is needed to understand and address the performance discrepancies in specific language pairs like German.

Summary updated

BibTeX
@inproceedings{shang2023hw,
  author = {Hengchao Shang and Zhiqiang Rao and Zongyao Li and Zhanglin Wu and Jiaxin Guo and Minghan Wang and Daimeng Wei and Shaojun Li and Zhengzhe Yu and Xiaoyu Chen and Lizhi Lei and Hao Yang},
  title = {{The HW-TSC’s Simultaneous Speech-to-Speech Translation System for IWSLT 2023 Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure