I2R’s End-to-End Speech Translation System for IWSLT 2023 Offline Shared Task

Muhammad Huzaifah, Kye Min Tan, Richeng DuanACL 2023

TL;DR

This paper presents I2R's submission to the IWSLT 2023 offline speech translation task, focusing on an end-to-end English-to-German system. They leverage large pretrained models and introduce novel pretraining and fine-tuning strategies, along with data augmentation and domain tagging, to achieve robust performance on challenging new test sets.

What Problem Does This Paper Address?

Traditionally, Speech Translation (ST) has been a cascaded process, combining Automatic Speech Recognition (ASR) with Machine Translation (MT). While this approach has been effective, it suffers from error propagation and higher latency. End-to-end (E2E) ST models, which directly translate speech to text, offer a more streamlined solution with lower latency and reduced error propagation. However, achieving performance comparable to cascaded systems, especially on diverse and challenging datasets, remains an active research area. This paper tackles the problem of building a high-performing E2E ST system for the IWSLT 2023 offline task, which includes new, more complex test sets beyond the standard TED talks, and operates under constrained data conditions.

Methods

The I2R system employs an end-to-end architecture with separate speech and text encoders, followed by a shared encoder and decoder. Key methodological contributions include:

  • Pretraining and Fine-tuning: The system is built upon large pretrained models like WavLM large for speech encoding and DeltaLM base for text encoding. This is followed by several stages of additional pretraining and fine-tuning to adapt the models to the downstream ST task.
  • Representation Mixing: To bridge the gap between speech and text modalities, they introduce a "representation mixing" strategy. This involves upsampling text tokens to match speech sequence lengths and then randomly swapping text embeddings with speech embeddings during training. This aims to create a joint multimodal representation.
  • Knowledge Distillation: To leverage the abundant text-only MT data available under the constrained track, they train a larger "teacher" MT model and use its outputs to guide the ST model's training via knowledge distillation.
  • Data Augmentation: On-the-fly audio augmentation techniques are applied to improve robustness against variations in audio recording quality.
  • Domain Tagging: The system incorporates domain tags ('<indomain>' and '<outdomain>') as prefix tokens during decoding to help the model adapt to different output styles and data distributions.
  • Model Ensemble: Multiple individual models are ensembled to further boost performance.

Datasets

The paper utilizes a variety of datasets for training and evaluation:

  • Text Data: A combination of in-domain (MuST-C v1/v2/v3, ST TED, TED2020) and out-of-domain (CoVOST v2, ELRC-CORDIS News, Europarl, OpenSubtitles, etc.) parallel corpora were used. A significant portion of the out-of-domain data was filtered using Language-agnostic BERT Sentence Embedding (LaBSE).
  • Audio Data: Both ASR corpora (Commonvoice, Librispeech, TED LIUM, Vox Populi) and ST corpora (CoVOST v2, Europarl-ST, MuST-C v3, ST TED) were used.
  • IWSLT 2023 Test Sets: The system is evaluated on the official IWSLT 2023 test sets, including the traditional TED talk evaluation set and two new, more challenging sets (ACL presentations, press conferences, and interviews).

Key Insights and Ideas

  • Pretrained Models are Crucial: The success of the system heavily relies on leveraging large-scale pretrained models, demonstrating their effectiveness in bridging the gap for ST tasks.
  • Modality Mixing is Beneficial: The representation mixing strategy, which combines speech and text inputs, appears to be a key factor in adapting the model to the downstream ST task, especially when the shared encoder is initially text-trained.
  • Domain Adaptation is Key for New Challenges: The introduction of new, more complex test sets in IWSLT 2023 necessitates explicit strategies like domain tagging to handle diverse data distributions and output styles.
  • Audio Augmentation Improves Robustness: Even for ST, on-the-fly audio augmentation proves beneficial, leading to a reduction in Word Error Rate (WER) in preliminary ASR experiments.

Key Results

  • The paper reports results on various test sets, with specific BLEU scores detailed in Tables 4 and 5. For the ST task (Table 5), their ensemble model achieved a BLEU score of 36.14 on MuST-C v3 and 36.93 on MuST-C v2 for the "in-domain" models.
  • The "extended-domain" models showed competitive performance, achieving 29.30 BLEU on CoVOST v2.
  • While the system performed well on many sets, they noted unexpectedly poor results on IWSLT-tst2019 and IWSLT-tst2020, which they attribute to potential weaknesses in the training procedure, domain mismatch, or resegmentation issues.

Limitations and Future Work

  • Computational Constraints: The authors acknowledge that time and computational constraints limited their exploration of certain avenues, such as training a multilingual model using unimodal ASR and MT data.
  • Resegmentation Issues: The poor performance on specific test sets (IWSLT-tst2019 and IWSLT-tst2020) is attributed, in part, to the lack of provided segmentation information and the use of an automatic segmentation tool (SHAS).
  • Future Work: The authors plan to investigate the reasons for the performance drop on specific test sets more thoroughly. They also express interest in further exploring the effect of modality mixing and improving models for speech-to-text tasks.

Summary updated

BibTeX
@inproceedings{huzaifah2023i,
  author = {Muhammad Huzaifah and Kye Min Tan and Richeng Duan},
  title = {{I2R’s End-to-End Speech Translation System for IWSLT 2023 Offline Shared Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure