Multilingual Speech Translation KIT @ IWSLT2021

Ngoc-Quan Pham, Tuan Nam Nguyen, Thanh-Le Ha, Sebastian Stüker, Alexander Waibel, Dan HeACL 2021

TL;DR

This paper presents a system for multilingual speech translation that combines traditional cascaded approaches with newer end-to-end models to achieve state-of-the-art results on the low-resource IWSLT2021 task. They demonstrate that enhancing Transformer architectures with relative attention and other improvements, and strategically combining cascaded and end-to-end models through ensembling, leads to significant gains in translation quality.

What Problem Does This Paper Address?

The paper tackles the challenge of multilingual speech translation (ST), particularly in low-resource settings. While automatic speech recognition (ASR) and machine translation (MT) have seen significant advancements, multilingual ST, which involves translating spoken language from one language to another, is more complex. Two main approaches exist: cascaded systems (ASR followed by MT) and end-to-end (E2E) systems. Cascaded systems benefit from separate, optimized components and larger datasets for each sub-task, but can suffer from error propagation. E2E systems theoretically avoid this but often require more data and can be harder to train. The IWSLT2021 Multilingual TEDx task, with its limited parallel data for ST, presents a perfect testbed for exploring how to best leverage these different approaches.

Methods

The authors developed a system that leverages both cascaded and E2E approaches.

  • Enhanced Transformer Architecture: They applied several enhancements to the standard Transformer architecture for all tasks (ASR, MT, and ST):
  • Relative Attention: This mechanism improves how the model attends to different parts of the input sequence by considering the relative positions of tokens, rather than just their absolute positions.
  • Dual Feed-Forward Networks (FFNs): Using two FFN layers per block instead of one, with a scaling factor of 0.5, helps in training deeper Transformers.
  • GELU and SiLU Activations: These activation functions, combined with gated linear units, are used to mitigate the "dying neuron" problem in deep networks.
  • Cascaded System: This involves a standard ASR system followed by an MT system.
  • End-to-End (E2E) System: This system directly translates speech to text in another language. They utilized pre-trained ASR encoders and then trained the E2E model.
  • Data Augmentation: To address the low-resource nature of the ST task, they employed data diversification techniques, including back-translation and model distillation, to generate synthetic parallel data.
  • Ensembling: The core of their strategy is to ensemble the outputs of the cascaded and E2E systems. This allows them to combine the strengths of both approaches, aiming for improved robustness and accuracy.

Datasets

The primary dataset used is the Multilingual TEDx corpus (Salesky et al., 2021). This corpus provides parallel data for speech recognition and machine translation across five languages: Spanish (es), French (fr), Italian (it), Portuguese (pt), and English (en). The paper notes that the speech translation parallel data is significantly scarcer than ASR or MT data.

Key Insights and Ideas

  • Cascaded vs. E2E Synergy: The paper strongly advocates for combining cascaded and E2E systems rather than viewing them as competing paradigms. Their ensembling approach demonstrates that this synergy can yield superior results, especially in low-resource scenarios.
  • Importance of Data Augmentation: In low-resource settings, synthetic data generation through techniques like data diversification is crucial for improving the performance of both MT and E2E ST models.
  • Transformer Enhancements Pay Off: The applied enhancements to the Transformer architecture, particularly relative attention, consistently improved performance across different tasks.
  • Error Propagation is a Real Challenge: The results highlight that while E2E models can theoretically avoid error propagation, they are still susceptible to data limitations. Conversely, cascaded systems, despite their potential for error propagation, can be very effective when components are well-optimized and data is abundant.

Key Results

  • Machine Translation: The introduction of relative attention and dual FFNs improved the baseline Transformer MT model by 0.95 BLEU points. Data diversification further boosted performance by nearly 1 BLEU point on average, with Italian-related pairs seeing up to a 2 BLEU point improvement.
  • Speech Recognition: The enhanced Transformer models achieved competitive results, with the fully enhanced version outperforming baseline models and LSTM-based systems. For example, on the Multilingual TEDx dataset, their ensemble achieved a Word Error Rate (WER) of 14.37 for es-en.
  • Speech Translation: The final ensemble of E2E and cascaded models achieved the best results. For instance, their primary submission (ensemble of E2E and Cascade with Syn+ESB) achieved a BLEU score of 31.02 for es-en, significantly outperforming the standalone E2E model (25.58 BLEU).

Limitations and Future Work

  • Ablation Studies: Due to time constraints, comprehensive ablation studies to isolate the exact contribution of each enhancement were not fully possible.
  • E2E Training Instability: Training E2E models for SLT was found to be unstable and prone to divergence, especially without ASR pre-training.
  • Zero-Shot Translation: The authors suggest investigating zero-shot translation for multilingual SLT as a future direction, acknowledging its difficulty due to modality differences.
  • Ensembling Scope: While their ensembling was effective, it was limited by the shared vocabulary between MT and SLT models and the averaging of output probabilities. Future work could explore more sophisticated ensembling strategies.

Summary updated

BibTeX
@inproceedings{pham2021multilingual,
  author = {Ngoc-Quan Pham and Tuan Nam Nguyen and Thanh-Le Ha and Sebastian Stüker and Alexander Waibel and Dan He},
  title = {{Multilingual Speech Translation KIT @ IWSLT2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure