ON-TRAC Consortium for End-to-End and Simultaneous Speech Translation Challenge Tasks at IWSLT 2020

Maha Elbayad, Ha Nguyen, Fethi Bougares, Natalia Tomashenko, Antoine Caubrière, Benjamin Lecouteux, Yannick Estève, Laurent Besacier

TL;DR

The ON-TRAC consortium details their high-performing submissions to the IWSLT 2020 challenge, demonstrating that robust data augmentation and model ensembling are key to state-of-the-art offline speech translation. For simultaneous translation, they introduce a "multi-path" training strategy for wait-k models that simplifies optimization while maintaining a competitive latency-quality trade-off.

What Problem Does This Paper Address?

Automatic Speech Translation (AST) is traditionally handled by "cascaded" systems that first transcribe speech to text (ASR) and then translate that text (MT). While effective, these cascades suffer from error propagation. Recent research has pivoted toward "end-to-end" models that translate audio directly to target text.

The authors tackle two specific challenges:

  1. Offline Translation: Can end-to-end models finally close the performance gap with traditional cascaded systems for English-to-German translation?
  2. Simultaneous Translation: How do we translate speech in real-time (online) with minimal delay (latency) without sacrificing too much accuracy (quality)? This requires the model to start translating before the speaker has even finished their sentence.

Methods

The consortium employed different architectures for the two tracks:

Offline Speech Translation

They used an attention-based encoder-decoder architecture trained end-to-end.

  • Encoder: A hybrid design featuring two VGG-like CNN blocks for initial feature extraction, followed by five Bidirectional LSTM (BLSTM) layers to capture long-range temporal context. Architecture of the speech encoder
  • Data Augmentation: This was a major focus. They used speed perturbation and SpecAugment (masking blocks of frequency and time in the spectrogram), which proved vital for performance.
  • Custom Segmentation: Instead of relying on the default (and sometimes noisy) dataset segments, they developed an ASR-based segmentation tool using Kaldi to better align audio chunks for the translation model.
Figure 1: Architecture of the speech encoder: a stack of two VGG blocks followed by 5 BLSTM layers.

Simultaneous Translation

For the low-latency track, they utilized Transformer-based wait-k decoders. In a wait-k policy, the model reads k source tokens before it starts writing the first target token, then alternates between reading one and writing one.

  • Unidirectional Encoders: Unlike offline models, they opted for unidirectional encoders. This avoids the massive computational cost of re-encoding the entire source prefix every time a new word is heard.
  • Multi-path Training: Instead of training a model for a specific k value, they proposed training a single model across an exhaustive set of wait-k paths. During training, a path is sampled uniformly, making the model more robust and flexible to different latency constraints at inference time.
  • ASR+MT Cascade: For the speech-to-text subtask, they paired a hybrid ASR system with their online MT system, using a custom algorithm to control the "lagging" between transcription and translation.

Datasets

The models were trained and evaluated on several standard benchmarks:

  • MuST-C: The primary English-German corpus.
  • Europarl-ST: A multilingual corpus of parliamentary debates.
  • How2: Used for synthetic data augmentation.
  • TED-LIUM 3: Used specifically for training the ASR component of the simultaneous cascade. Statistics of training and evaluation data
Statistics of training and evaluation data

Key Insights and Ideas

  • SpecAugment is Non-Negotiable: The authors found that applying SpecAugment was the single most impactful factor in outperforming their baseline offline models.
  • Segmentation Matters: The quality of the initial audio segmentation significantly affects BLEU scores. Their custom ASR-based segmentation outperformed the provided defaults, especially on noisy data.
  • The Power of Multi-path: Multi-path training allows a single model to perform well across various latency settings. It eliminates the need to painstakingly tune the k parameter during the training phase, which is a significant practical advantage.
  • Ensembling Still Reigns: Even in the era of massive end-to-end models, ensembling multiple diverse models (e.g., those trained on different data combinations) provided a consistent and significant boost in translation quality.

Key Results

  • Offline Track: Their primary system (an ensemble of six models) achieved the highest scores, reaching a BLEU of 25.22 on the MuST-C common test set. BLEU scores for offline experiments
  • Simultaneous Text-to-Text: Ensembling wait-k models provided an average boost of 1.43 BLEU points. The multi-path model proved highly competitive with specialized models optimized for a single latency. Text-to-Text Latency-quality trade-offs
  • Simultaneous Speech-to-Text: The ASR+MT cascade achieved a strong latency-quality trade-off, with ensembling again providing a 0.67 BLEU point improvement. Speech-to-Text Latency-quality trade-offs
BLEU scores for offline experiments
Text-to-Text Latency-quality trade-offs
Speech-to-Text Latency-quality trade-offs

Limitations and Future Work

While the offline results for end-to-end models are encouraging, the simultaneous speech-to-text system still relies on a cascaded ASR+MT approach. The authors acknowledge that the next major step is moving toward fully end-to-end simultaneous speech translation, which would eliminate the intermediate text transcription step and potentially further reduce latency and error propagation. Additionally, the current latency control for the cascade requires manual tuning of hyperparameters on a development set, suggesting a need for more automated or adaptive latency management.

Summary updated

Abstract

This paper describes the ON-TRAC Consortium translation systems developed for two challenge tracks featured in the Evaluation Campaign of IWSLT 2020, offline speech translation and simultaneous speech translation. ON-TRAC Consortium is composed of researchers from three French academic laboratories: LIA (Avignon Université), LIG (Université Grenoble Alpes), and LIUM (Le Mans Université). Attention-based encoder-decoder models, trained end-to-end, were used for our submissions to the offline speech translation track. Our contributions focused on data augmentation and ensembling of multiple models. In the simultaneous speech translation track, we build on Transformer-based wait-k models for the text-to-text subtask. For speech-to-text simultaneous translation, we attach a wait-k MT system to a hybrid ASR system. We propose an algorithm to control the latency of the ASR+MT cascade and achieve a good latency-quality trade-off on both subtasks.
BibTeX
@inproceedings{elbayad2020trac,
  author = {Maha Elbayad and Ha Nguyen and Fethi Bougares and Natalia Tomashenko and Antoine Caubrière and Benjamin Lecouteux and Yannick Estève and Laurent Besacier},
  title = {{ON-TRAC Consortium for End-to-End and Simultaneous Speech Translation Challenge Tasks at IWSLT 2020}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}

Loading related content...

Figure