MLLP-VRAIN UPV systems for the IWSLT 2022 Simultaneous Speech Translation and Speech-to-Speech Translation tasks

Javier Iranzo-Sánchez, Javier Jorge Cano, Alejandro Pérez-González-de-Martos, Adrián Giménez Pastor, Gonçal V. Garcés Díaz-Munío, Pau Baquero-Arnal, Joan Albert Silvestre-Cerdà, Jorge Civera Saiz, Albert Sanchis, Alfons JuanACL 2022

TL;DR

This paper details the MLLP-VRAIN research group's participation in the IWSLT 2022 conference, presenting cascaded systems for both simultaneous speech translation (SST) and speech-to-speech translation (S2ST) from English to German. They highlight a novel strategy for achieving low-latency SST by incorporating unconsolidated ASR hypotheses into the MT system, and introduce a speaker-adaptive TTS module for S2ST.

What Problem Does This Paper Address?

The paper tackles the challenge of building robust and efficient systems for real-time speech translation. Specifically, it addresses two key areas:

  1. Simultaneous Speech Translation (SST): This involves translating spoken language as it is being produced, requiring a delicate balance between low latency and translation quality. Traditional approaches often struggle to minimize the delay without sacrificing accuracy.
  2. Speech-to-Speech Translation (S2ST): This task requires not only translating spoken language but also synthesizing the translated speech in a natural-sounding voice. A key challenge here is achieving speaker adaptation, where the synthesized speech mimics the characteristics of the original speaker.

The authors aim to develop systems that can perform these tasks effectively, paying close attention to data preparation, model architecture, and the integration of individual components within a cascaded framework.

Methods

The MLLP-VRAIN team employed a cascaded approach, building upon individual Automatic Speech Recognition (ASR), Machine Translation (MT), and Text-to-Speech (TTS) components.

  • ASR System: For SST, they used a chunk-based BLSTM acoustic model (AM) combined with a Transformer Language Model (TLM). For S2ST, they used a less restrictive pruning setup for the ASR system, allowing for more look-ahead context.
  • MT System: For SST, a multi-k Transformer-based MT system was used. For S2ST, the offline MT system was employed. A significant focus was placed on data preparation, including filtering and upsampling techniques.
  • Simultaneous Translation Strategy: A novel strategy for SST was developed: when a new word is consolidated by the ASR, the unconsolidated part of the top-scoring hypothesis is also sent to the MT system. This aims to reduce latency by leveraging potentially stable but not yet fully confirmed ASR outputs.
  • TTS System (for S2ST): A speaker-adaptive TTS module was developed. This system utilizes a transfer learning approach, leveraging a modified ResNet-34 speaker encoder trained on a speaker classification task. This encoder is used to compute speaker embeddings, allowing the TTS model to mimic the voice characteristics of the original speaker. The TTS pipeline consists of a Conformer-based text-to-spectrogram network and a multi-band UnivNet neural vocoder.

Datasets

The following datasets were used for training and evaluation:

  • ASR Training: 3649 hours of transcribed speech from various sources including Common Voice 6.1, Librispeech, MuST-C v2.0, How2, and Europarl-ST v1.1.
  • MT Training: A combination of datasets including WMT20/WMT21 news translation data, Europarl-ST, MuST-C v2, TED corpus, and OpenSubtitles. Data filtering and cleaning techniques were applied.
  • LM Training: Large text corpora including News discussions, News crawl, Open Subs, WikiMatrix, UN Parallel Corpus, Europarl, News Commentary, Librispeech, Common Voice, MuST-C, How2, Europarl-ST, WIT3.
  • TTS Training: TED-LIUM v3 dataset (for speaker encoder) and LibriVoxDeEn dataset (for text-to-spectrogram model).
  • Evaluation: MuST-C v2.0 dev and test sets (tst-HE, tst-COMMON), IWSLT 2015, and IWSLT 2018 test sets.

Key Insights and Ideas

  • Leveraging Unconsolidated ASR Hypotheses for Low Latency: The strategy of feeding unconsolidated ASR hypotheses to the MT system proved effective in achieving very low latency (below AL=1000) for SST, albeit with a small degradation in translation quality. This highlights the potential of exploiting partial information in cascaded systems.
  • Speaker Adaptation in TTS: The use of a transfer learning approach with a pre-trained speaker encoder enabled effective zero-shot speaker adaptation in the TTS system, allowing it to mimic target speaker characteristics without extensive speaker-specific training data.
  • Data Preparation is Crucial: The authors emphasize the significant effort invested in data preparation, selection, and filtering for the MT system, indicating its critical role in achieving good performance.
  • Trade-off Between Latency and Quality: The paper consistently demonstrates the inherent trade-off between latency and translation quality in SST, and the proposed methods aim to navigate this trade-off effectively.

Key Results

  • SST: The proposed strategy of using unconsolidated ASR hypotheses allowed the system to stay below an Average Latency (AL) threshold of 1000 ms on the MuST-C.v2 tst-COMMON dataset. System #8 (using data upsampling and OpenSubtitles) was submitted as the primary system.
  • MT Quality: The multi-k Transformer MT system, when trained on various data setups, achieved competitive BLEU scores. System #3 (Baseline + OpenSubtitles) and System #8 (Clean ups. & OpenSubtitles) were selected for further experimentation.
  • ASR Evaluation: Using a Transformer LM (TLM) instead of a 4-gram model roughly halved perplexity and led to a consistent WER reduction of about 23-25%. Increasing look-ahead context in the ASR improved accuracy but increased latency.

Limitations and Future Work

  • Latency-Quality Trade-off: While the unconsolidated ASR hypothesis strategy improved latency, it did result in a small degradation of translation quality.
  • Hyperparameter Optimization: The authors acknowledge that significant hyperparameter optimization is still required to properly integrate the different components of the cascade system.
  • Future Work: The paper suggests further research into the use of partial hypotheses by the MT system and other downstream components to further improve the quality-latency trade-off in SST.

Summary updated

BibTeX
@inproceedings{iranzosanchez2022mllp,
  author = {Javier Iranzo-Sánchez and Javier Jorge Cano and Alejandro Pérez-González-de-Martos and Adrián Giménez Pastor and Gonçal V. Garcés Díaz-Munío and Pau Baquero-Arnal and Joan Albert Silvestre-Cerdà and Jorge Civera Saiz and Albert Sanchis and Alfons Juan},
  title = {{MLLP-VRAIN UPV systems for the IWSLT 2022 Simultaneous Speech Translation and Speech-to-Speech Translation tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure