Edinburgh's End-to-End Multilingual Speech Translation System for IWSLT 2021

Biao Zhang, Biao Zhang 0006, Rico SennrichIwslt 2021 · DOI

TL;DR

This paper presents Edinburgh's end-to-end multilingual speech translation (ST) system for IWSLT 2021, achieving competitive results by integrating advanced Transformer components and novel techniques like adaptive feature selection and data augmentation. The work demonstrates significant improvements in low-resource ST, particularly in zero-shot translation, by effectively leveraging transfer learning across languages.

What Problem Does This Paper Address?

End-to-end (E2E) speech translation (ST) has shown great promise, often outperforming traditional cascaded systems. However, a major bottleneck for E2E ST is the scarcity of dedicated parallel speech-to-translation training data. While multilingual machine translation (MT) has successfully addressed data scarcity through transfer learning and zero-shot translation, applying these benefits to E2E ST, especially in low-resource scenarios, remains an open challenge. The IWSLT 2021 task specifically targets this by focusing on low-resource language pairs and including a zero-shot condition where only ASR training data is available.

Methods

The authors build their E2E multilingual ST system on the Transformer architecture, incorporating several key techniques:

  • Adaptive Feature Selection (AFS): This method prunes uninformative speech features from the audio input, reducing redundancy and improving ST performance. It involves pretraining an ASR encoder-decoder, then finetuning the ASR model with AFS, and finally freezing these components for ST training.
  • Deep and Big Transformer: To increase modeling capacity, they employ depth-scaled initialization (DS-Init) to train deeper Transformers and use a "Transformer-big" architecture for the ST model.
  • Language-Specific Modeling: To mitigate task interference in multilingual settings, they introduce language-specific linear mappings in both the ASR and ST encoders, grounded by language embeddings.
  • Sparsified Linear Attention (ReLA): Applied to the ST decoder, ReLA enforces sparsity in attention logits, improving computational efficiency without sacrificing performance.
  • Root Mean Square Layer Normalization (RMSNorm): This replaces standard LayerNorm in the ST encoder and decoder, offering a more efficient alternative that stabilizes training for deep models.
  • Data Augmentation (DA): They leverage knowledge distillation from a multilingual MT model to generate pseudo-parallel ST data, effectively converting the zero-shot problem into a zero-resource one.
  • Multi-Task Learning: The system also incorporates ASR and auto-encoding tasks to further enhance transfer learning.

Datasets

The primary dataset used is Multilingual TEDx (Salesky et al., 2021). This dataset contains audio from TEDx talks in 8 source languages, paired with manual transcriptions, and covers translations into 5 target languages. For the IWSLT 2021 task, they focus on 4 source languages (Es, Fr, Pt, It) with final evaluation on translations into En and Es. The dataset is characterized by its small scale, with ST training data ranging from 5K to 39K utterances per direction, making it ideal for studying transfer learning.

Key Insights and Ideas

  • Transfer Learning is Crucial for Low-Resource ST: The paper strongly emphasizes that multilingual modeling and transfer learning are vital for improving performance in low-resource ST scenarios, especially for zero-shot directions.
  • Data Augmentation is a Game-Changer: Generating pseudo-parallel data via knowledge distillation from MT models significantly boosts performance, particularly for zero-shot translation, highlighting the adage "data matter!"
  • Component Synergy: The various techniques employed (AFS, deep/big models, language-specific modeling, ReLA, RMSNorm, DA) are complementary and collectively contribute to substantial improvements.
  • E2E Outperforms Cascading: The E2E multilingual ST system consistently outperforms their own cascading system, demonstrating the advantages of the end-to-end paradigm when properly optimized.
  • Deep Models Can Work for ST: While deep models can be challenging for ST, techniques like DS-Init and RMSNorm enable their effective use, leading to improved quality.

Key Results

  • The best multilingual MT model (24 layers) surpassed the official baseline by over 8 BLEU.
  • The E2E multilingual ST system achieved an average BLEU of 29.61 with a base 6-layer model, outperforming the official baseline by ~17 BLEU.
  • The final submission, an ensemble of 6 E2E models, achieved an average BLEU of 31.34, securing runner-up position in the IWSLT 2021 task.
  • The system significantly improved zero-shot translation performance, with data augmentation contributing an additional ~7.54 BLEU on these directions.
  • The E2E system also outperformed their cascading counterpart by 2.48 average BLEU.
  • Multilingual ASR performance was also strong, achieving an average WER of 39.49, significantly outperforming the official baseline.

Limitations and Future Work

  • Generalization to Distant Languages: The current work primarily studies transfer across similar languages. Exploring how the model generalizes to linguistically distant languages remains an open question.
  • Effectiveness of Deep Models: While deep models were shown to be effective with specific techniques, the paper notes that deep ST models seem to be ineffective in general. Further investigation into the reasons for this and potential solutions is suggested.
  • Cascading System Sub-optimality: The authors acknowledge that their cascading system was sub-optimal, suggesting that further improvements in avoiding bias towards ASR outputs could enhance its performance.

Summary updated

BibTeX
@inproceedings{zhang2021edinburgh,
  author = {Biao Zhang and Biao Zhang and Rico Sennrich},
  title = {{Edinburgh's End-to-End Multilingual Speech Translation System for IWSLT 2021}},
  booktitle = {Iwslt},
  year = {2021},
  doi = {10.18653/v1/2021.iwslt-1.19},
}
Figure