ON-TRAC Consortium for End-to-End and Simultaneous Speech Translation Challenge Tasks at IWSLT 2020

Maha Elbayad, Ha Nguyen, Fethi Bougares, Natalia Tomashenko, Antoine Caubrière, Benjamin Lecouteux, Yannick Estève, Laurent Besacier

TL;DR

This paper details the ON-TRAC consortium's participation in the IWSLT 2020 challenge, presenting end-to-end models for offline speech translation and novel cascaded systems for simultaneous speech translation. They demonstrate the effectiveness of data augmentation, model ensembling, and a carefully tuned ASR-MT cascade for achieving a good latency-quality trade-off.

What Problem Does This Paper Address?

The paper tackles two significant challenges in speech translation:

  1. Offline Speech Translation: The goal is to translate spoken language into text without real-time constraints. While cascaded systems (ASR + MT) have been state-of-the-art, there's a growing interest in end-to-end models that directly map speech to text in the target language. The IWSLT 2020 offline track specifically aimed to evaluate if these end-to-end systems could close the gap with cascaded approaches for the English-to-German pair.
  2. Simultaneous Speech Translation: This is a much more challenging task where translation must be generated before the entire source speech utterance is available. This introduces a critical latency constraint, requiring a delicate balance between translation quality and how quickly the output is produced. Previous work on low-latency neural speech translation is limited, making this a key area for research.

Methods

The ON-TRAC consortium employed several key methods:

  • End-to-End Offline Speech Translation:
    • Architecture: They used attention-based encoder-decoder models, specifically adapting their previous year's architecture. The encoder consists of VGG-like CNN blocks followed by BLSTM layers, and the decoder uses LSTM layers.
  • Data Augmentation: Techniques like speed perturbation and SpecAugment (time warping, frequency masking, time masking) were crucial for improving performance, especially when combining different corpora.
  • Model Ensembling: Combining predictions from multiple trained models significantly boosted BLEU scores across all test sets.
  • Simultaneous Speech Translation:
    • Cascaded ASR + MT System: This involved an automatic speech recognition (ASR) system followed by an online machine translation (MT) system.
  • Online MT (Wait-k Models): They utilized Transformer-based wait-k decoders with unidirectional encoders. The "wait-k" policy means the model reads k source tokens before starting to generate the target. They explored optimizing models with a single wait-k path and also a "multi-path" approach that jointly optimizes across multiple wait-k paths for richer training signals.
  • ASR Endpointing: The latency of the cascaded system was primarily controlled by the ASR system's endpointing mechanism, which decides when to output a partial transcription. They used Kaldi for ASR training and employed specific rules to detect utterance boundaries.
  • Latency Control: For the MT system, they constrained the number of tokens it could decode based on the ASR output and hyper-parameters (α, β) to manage the overall latency.

Datasets

  • Training Data:
    • MuST-C: English-to-German corpus (original and synthetically augmented versions).
  • Europarl: English-to-German corpus.
  • How2: English-to-German synthetic corpus.
  • TED-LIUM 3: Used for ASR training.
  • Common Crawl, News Commentary: Used for MT training.
  • Evaluation Data:
    • IWSLT 2020 test sets: Specifically tst-COMMON, tst-HE, tst2019, and tst2020.
  • MuST-C tst-COMMON: Used for evaluating simultaneous translation.

Key Insights and Ideas

  • Data Augmentation is Key: SpecAugment proved particularly effective, especially when combining diverse corpora, leading to significant improvements on specific test sets.
  • Ensembling for Robustness: Ensembling multiple models consistently yielded the best results for offline translation, highlighting the benefit of diverse model perspectives.
  • Custom ASR Segmentation is Better: Their automatically generated ASR segmentation improved results over the provided segmentation, suggesting that noisy or imperfect segmentation can hinder performance.
  • Unidirectional Encoders for Online MT: Using unidirectional encoders in their wait-k MT models alleviated re-encoding costs and achieved a better quality-lagging trade-off compared to bidirectional encoders in the online setting.
  • Multi-path Training for MT: Jointly optimizing across multiple wait-k paths provided a richer training signal and competitive performance, sometimes outperforming single-path wait-k models.
  • Controlled Latency via ASR Endpointing: The ASR system's endpointing rules were the primary mechanism for controlling the latency of the cascaded speech-to-text system, allowing for a tunable trade-off.

Key Results

  • Offline Translation: The ensembled models achieved the best BLEU scores across all test sets, with the ensemble of six models reaching 25.22 BLEU on tst-COMMON and 23.80 on tst-HE.
  • Simultaneous Translation:
    • Text-to-Text: Their multi-path approach and ensembling of wait-k models showed competitive latency-quality trade-offs, with ensembling boosting BLEU by 1.43 points on average.
  • Speech-to-Text: The cascaded ASR+MT system, with a single ASR model and ensembled MT wait-k models, followed similar trends, with ensembling boosting BLEU by 0.67 points on average.
  • ASR Performance: Offline ASR achieved a WER of 14.2% on MuST-C tst-COMMON, while the online ASR had a slightly higher WER of 16.3% (with an AL of 5s).

Limitations and Future Work

  • Limitations:
    • The paper notes that adding MuST-C synthetic data to certain pools surprisingly decreased BLEU scores on some test sets, and they couldn't fully investigate this due to time constraints.
  • The impact of fine-tuning on specific models was found to be very limited.
  • The performance of the cascaded ASR+MT system was still lower than offline systems, as expected due to the latency constraints.
  • Future Work: The authors explicitly state that future work will be dedicated to simultaneous speech translation using end-to-end models, moving away from cascaded approaches.

Summary updated

Abstract

This paper describes the ON-TRAC Consortium translation systems developed for two challenge tracks featured in the Evaluation Campaign of IWSLT 2020, offline speech translation and simultaneous speech translation. ON-TRAC Consortium is composed of researchers from three French academic laboratories: LIA (Avignon Université), LIG (Université Grenoble Alpes), and LIUM (Le Mans Université). Attention-based encoder-decoder models, trained end-to-end, were used for our submissions to the offline speech translation track. Our contributions focused on data augmentation and ensembling of multiple models. In the simultaneous speech translation track, we build on Transformer-based wait-k models for the text-to-text subtask. For speech-to-text simultaneous translation, we attach a wait-k MT system to a hybrid ASR system. We propose an algorithm to control the latency of the ASR+MT cascade and achieve a good latency-quality trade-off on both subtasks.
BibTeX
@inproceedings{elbayad2020trac,
  author = {Maha Elbayad and Ha Nguyen and Fethi Bougares and Natalia Tomashenko and Antoine Caubrière and Benjamin Lecouteux and Yannick Estève and Laurent Besacier},
  title = {{ON-TRAC Consortium for End-to-End and Simultaneous Speech Translation Challenge Tasks at IWSLT 2020}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure