ZJU’s IWSLT 2021 Speech Translation System

Linlin ZhangACL 2021

TL;DR

This paper presents ZJU's submission to the IWSLT 2021 Multilingual Speech Translation Task, focusing on building robust speech translation (ST) systems for non-English languages. They explore both cascaded (ASR + NMT) and end-to-end approaches, finding that an ensembled cascaded system achieved the best results, particularly for low-resource language pairs.

What Problem Does This Paper Address?

Speech translation (ST) is a challenging task that involves converting spoken language from one language to another. The IWSLT 2021 Multilingual Speech Translation Task specifically focuses on ST across many non-English source languages, highlighting the need for effective methods that can leverage limited parallel data, especially for low-resource languages. The task also presents a "zero-shot" condition, where models must translate between language pairs for which no direct training data exists. This necessitates approaches that can effectively transfer knowledge and generalize to unseen language combinations.

Methods

The authors explored two primary architectures:

  • Cascaded Speech Translation: This approach breaks down ST into two distinct stages:
  • Automatic Speech Recognition (ASR): They employed Conformer models, a hybrid architecture combining convolutional and self-attention mechanisms, for ASR. These models were trained on provided speech and transcripts.
  • Machine Translation (MT): A unified Transformer model was used for multilingual neural machine translation (NMT). To improve robustness, they incorporated noisy ASR transcripts, used back-translation (BT) to generate synthetic parallel data, and applied a "mask trick" similar to BERT for training. They also leveraged reverse language pairs (e.g., Spanish-to-Italian to help Italian-to-Spanish) and BT for zero-shot translation.
  • End-to-End Direct Speech Translation: This approach directly maps speech to text in the target language using a single Transformer-based model. Key techniques included:
  • Data Augmentation: Applying audio effects like tempo and speed changes using Sox.
  • Multitask Learning: Incorporating an auxiliary ASR task during training to improve ST performance. This involved sharing encoder and decoder layers and using a joint loss function.
  • Pre-training: Utilizing pre-trained ASR models to initialize the ST model.

Finally, their best-performing system was an ensemble of cascaded models.

Datasets

The primary dataset used was the IWSLT2021 Multilingual Speech Translation Task dataset. This dataset includes:

  • Speech and transcripts for four source languages: Spanish, French, Portuguese, and Italian.
  • Translations in a subset of five languages: English, Spanish, French, Portuguese, and Italian.

The task provided data under two conditions: supervised and zero-shot. The authors also mention using the Multilingual TEDx dataset for training their cascaded MT models.

Key Insights and Ideas

  • Cascaded systems excel for low-resource pairs: The paper demonstrates that cascaded ASR+NMT systems, especially when ensembled, provide superior performance for low-resource language pairs compared to end-to-end models.
  • Multilingual NMT is crucial for zero-shot: Using a single, unified Transformer model for all language pairs in the NMT component, along with techniques like reverse language pairs and back-translation, proved effective for tackling zero-shot translation scenarios.
  • Pre-training ASR decoder is not beneficial for ST: Surprisingly, pre-training the ASR decoder in the multitask end-to-end setup did not improve ST performance, though it did reduce ASR loss and training time.
  • Single-language ASR pre-training is better than multilingual: For end-to-end ST, pre-training with ASR models trained on a single language (Spanish) yielded better results than pre-training with a model trained on all four languages. This suggests that richer ASR corpora don't always translate to better ST performance in this context.
  • Conformer ASR shows promise: The Conformer model demonstrated strong performance for ASR, outperforming standard Transformer models, especially with more encoder layers.

Key Results

  • The submitted system was an ensembled cascaded system.
  • The paper reports BLEU scores for various language pairs and model configurations. For instance, the ensembled cascaded system achieved scores like 34.5 on ES-EN and 20.8 on IT-EN.
  • Table 4 shows that cascaded models generally outperformed end-to-end models, particularly for language pairs like IT-EN and IT-ES.
  • End-to-end models showed promise in closing the performance gap for high-resource language pairs.

Limitations and Future Work

  • End-to-end vs. Cascaded: While cascaded systems performed better overall, especially for low-resource pairs, end-to-end models are closing the gap for high-resource pairs, suggesting future work could focus on improving their performance in low-resource settings.
  • Multilingual Pre-training Surprise: The finding that single-language ASR pre-training was better than multilingual pre-training for end-to-end ST is unexpected and warrants further investigation into why this occurs and how to leverage multilingual ASR more effectively.
  • Ensembling: The success of the ensembled cascaded system indicates that exploring more sophisticated ensembling techniques could further boost performance.
  • Parameter Tuning: The paper mentions trying various parameter settings, but a more systematic exploration of hyperparameter optimization could yield further improvements.

Summary updated

BibTeX
@inproceedings{zhang2021zjus,
  author = {Linlin Zhang},
  title = {{ZJU’s IWSLT 2021 Speech Translation System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure