The BIGAI Offline Speech Translation Systems for IWSLT 2023 Evaluation

Zhihang XieACL 2023

TL;DR

This paper presents BIGAI's end-to-end speech translation systems for the IWSLT 2023 offline task, leveraging Wav2Vec2 for speech recognition and mBART50 for machine translation. They introduce a three-stage fine-tuning strategy with CTC loss to bridge the speech and translation modules, achieving competitive BLEU scores across English-to-Chinese, German, and Japanese tracks, though performance dips in challenging scenarios like presentations.

What Problem Does This Paper Address?

Speech translation, the task of converting spoken language from one language to another, is a complex challenge. Traditional approaches often use a cascaded system: first, an Automatic Speech Recognition (ASR) system transcribes the audio, and then a Machine Translation (MT) system translates the text. While this allows leveraging state-of-the-art ASR and MT models, it suffers from error propagation and the loss of paralinguistic cues. End-to-end (E2E) systems aim to directly translate speech to text, potentially capturing more information and simplifying the pipeline. However, a significant hurdle for E2E systems is the scarcity of high-quality, large-scale parallel speech-to-text translation data. This paper tackles this challenge by exploring techniques to improve E2E speech translation performance, particularly in an offline setting where real-time processing is not a constraint.

Methods

The BIGAI team employs a three-stage fine-tuning strategy for their E2E speech translation systems.

  1. Pretrained Models: They start with powerful, self-supervised pretrained models: wav2vec2-large-960h-lv60-self for speech recognition and mbart-large-50-one-to-many-mmt for machine translation.
  2. Speech Recognition Fine-tuning: The Wav2Vec2 model is fine-tuned on large speech corpora (LibriSpeech, Common Voice, MuSTC, TEDLIUM, VoxPopuli) to create an ASR model (referred to as ASR12).
  3. Machine Translation Fine-tuning: The mBART50 model is fine-tuned on parallel text translation datasets (MuSTC, OpenSubtitles, News Commentaries) for each target language (German, Japanese, Chinese).
  4. End-to-End Speech Translation Fine-tuning: This is the core of their approach. They assemble the fine-tuned ASR and MT models into an E2E system. An adapter module is introduced to bridge the speech and translation modules. Crucially, they incorporate Connectionist Temporal Classification (CTC) loss during training. This CTC loss leverages the divergence between speech features and source token sequences, aiming to improve the alignment and translation quality. The training loss is a combination of CTC loss for ASR and cross-entropy loss for MT, with a hyperparameter to balance their weights.

They also implement a speech resegmentation technique to handle long audio clips by splitting them into shorter segments and then merging consecutive segments, aiming to optimize segment duration for better translation.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Speech Corpora (for ASR training): LibriSpeech, Common Voice v11.0, MuSTC (en→de, en→ja, en→zh), TEDLIUM v3, and VoxPopuli.
  • Translation Corpora (for MT training): MuSTC (en→de, en→ja, en→zh), OpenSubtitles v2018, and News Commentaries v16.
  • Evaluation Datasets:
    • tst2023: The primary evaluation benchmark for the IWSLT 2023 offline task, covering English-to-German, English-to-Japanese, and English-to-Chinese. This dataset is noted to contain more challenging scenarios like presentations and interviews.
  • tst-COMMON: Used for sentence-level translation evaluation.
  • Past years' IWSLT test datasets: Used for corpus-level translation evaluation, particularly to assess the impact of speech resegmentation.

Key Insights and Ideas

  • Bridging Speech and Text with Adapters and CTC: The introduction of an adapter module and the integration of CTC loss are key to effectively connecting the speech recognition and machine translation components in an end-to-end fashion. This suggests that explicitly modeling the speech-to-token alignment during translation training is beneficial.
  • Three-Stage Fine-tuning is Effective: The staged approach, starting with ASR and MT fine-tuning before the full E2E training, allows for leveraging strong unimodal models and then progressively building the translation system.
  • Speech Resegmentation Matters for Challenging Scenarios: The experiments on past IWSLT datasets highlight that optimizing segment duration (e.g., 20s to 90s) significantly improves BLEU scores, suggesting that longer, more contextual segments are better for translation, especially when avoiding "brutal" segmentation.
  • Performance Degradation in Complex Scenarios: The results on the tst2023 dataset reveal a significant drop in performance for challenging audio types like presentations and interviews. This indicates that current E2E systems struggle with spontaneous speech, non-native speakers, and complex acoustic environments.

Key Results

  • BLEU Scores on tst2023: The proposed PT36 models achieved BLEU scores of 22.3 (en→de), 10.7 (en→ja), and 33.0 (en→zh) on the tst2023 datasets.
  • MT Improvement: Fine-tuning MT models with out-of-domain corpora (OpenSubtitles, NewsCommentaries) improved BLEU scores by up to 16.1% compared to base MT models.
  • E2E vs. MT: The E2E PT36 models showed a performance degradation compared to fine-tuned MT models, with BLEU scores dropping by approximately 16-20% across the language pairs.
  • Speech Resegmentation Impact: Optimizing segment duration (20s-90s) led to an average BLEU score increase of about 18.7% on past test datasets.
  • ASR Performance: The ASR12 model achieved a WER of 34.73% on the MuSTC datasets, which is noted as being significantly worse than other contemporary systems.

Limitations and Future Work

  • ASR Module Weakness: The paper acknowledges that the ASR module's performance is not "good enough," which negatively impacts the overall E2E system. This is partly attributed to the large vocabulary size and insufficient training time.
  • Challenging Scenarios: The systems exhibit inadequate abilities in handling non-native speakers, different accents, spontaneous speech, and controlled interaction with a second speaker, as evidenced by the performance drop on presentations and interviews.
  • Limited Resources: The authors mention that some modules were not "well trained" due to limited resources, suggesting that more extensive training could yield better results.
  • Future Work: The paper implicitly suggests that further improvements in ASR robustness, better handling of complex acoustic and linguistic scenarios, and more extensive training on diverse data are crucial for advancing E2E speech translation systems to outperform SOTA cascaded systems.

Summary updated

BibTeX
@inproceedings{xie2023bigai,
  author = {Zhihang Xie},
  title = {{The BIGAI Offline Speech Translation Systems for IWSLT 2023 Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure