TL;DR
This paper details the ON-TRAC consortium's participation in the IWSLT 2020 challenge, presenting end-to-end models for offline speech translation and novel cascaded systems for simultaneous speech translation. They demonstrate the effectiveness of data augmentation, model ensembling, and a carefully tuned ASR-MT cascade for achieving a good latency-quality trade-off.
What Problem Does This Paper Address?
The paper tackles two significant challenges in speech translation:
- Offline Speech Translation: The goal is to translate spoken language into text without real-time constraints. While cascaded systems (ASR + MT) have been state-of-the-art, there's a growing interest in end-to-end models that directly map speech to text in the target language. The IWSLT 2020 offline track specifically aimed to evaluate if these end-to-end systems could close the gap with cascaded approaches for the English-to-German pair.
- Simultaneous Speech Translation: This is a much more challenging task where translation must be generated before the entire source speech utterance is available. This introduces a critical latency constraint, requiring a delicate balance between translation quality and how quickly the output is produced. Previous work on low-latency neural speech translation is limited, making this a key area for research.
Methods
The ON-TRAC consortium employed several key methods:
- End-to-End Offline Speech Translation:
- Architecture: They used attention-based encoder-decoder models, specifically adapting their previous year's architecture. The encoder consists of VGG-like CNN blocks followed by BLSTM layers, and the decoder uses LSTM layers.
- Data Augmentation: Techniques like speed perturbation and SpecAugment (time warping, frequency masking, time masking) were crucial for improving performance, especially when combining different corpora.
- Model Ensembling: Combining predictions from multiple trained models significantly boosted BLEU scores across all test sets.
- Simultaneous Speech Translation:
- Cascaded ASR + MT System: This involved an automatic speech recognition (ASR) system followed by an online machine translation (MT) system.
- Online MT (Wait-k Models): They utilized Transformer-based wait-k decoders with unidirectional encoders. The "wait-k" policy means the model reads
ksource tokens before starting to generate the target. They explored optimizing models with a single wait-k path and also a "multi-path" approach that jointly optimizes across multiple wait-k paths for richer training signals. - ASR Endpointing: The latency of the cascaded system was primarily controlled by the ASR system's endpointing mechanism, which decides when to output a partial transcription. They used Kaldi for ASR training and employed specific rules to detect utterance boundaries.
- Latency Control: For the MT system, they constrained the number of tokens it could decode based on the ASR output and hyper-parameters (
α,β) to manage the overall latency.
Datasets
- Training Data:
- MuST-C: English-to-German corpus (original and synthetically augmented versions).
- Europarl: English-to-German corpus.
- How2: English-to-German synthetic corpus.
- TED-LIUM 3: Used for ASR training.
- Common Crawl, News Commentary: Used for MT training.
- Evaluation Data:
- IWSLT 2020 test sets: Specifically
tst-COMMON,tst-HE,tst2019, andtst2020.
- IWSLT 2020 test sets: Specifically
- MuST-C tst-COMMON: Used for evaluating simultaneous translation.
Key Insights and Ideas
- Data Augmentation is Key: SpecAugment proved particularly effective, especially when combining diverse corpora, leading to significant improvements on specific test sets.
- Ensembling for Robustness: Ensembling multiple models consistently yielded the best results for offline translation, highlighting the benefit of diverse model perspectives.
- Custom ASR Segmentation is Better: Their automatically generated ASR segmentation improved results over the provided segmentation, suggesting that noisy or imperfect segmentation can hinder performance.
- Unidirectional Encoders for Online MT: Using unidirectional encoders in their wait-k MT models alleviated re-encoding costs and achieved a better quality-lagging trade-off compared to bidirectional encoders in the online setting.
- Multi-path Training for MT: Jointly optimizing across multiple wait-k paths provided a richer training signal and competitive performance, sometimes outperforming single-path wait-k models.
- Controlled Latency via ASR Endpointing: The ASR system's endpointing rules were the primary mechanism for controlling the latency of the cascaded speech-to-text system, allowing for a tunable trade-off.
Key Results
- Offline Translation: The ensembled models achieved the best BLEU scores across all test sets, with the ensemble of six models reaching 25.22 BLEU on
tst-COMMONand 23.80 ontst-HE. - Simultaneous Translation:
- Text-to-Text: Their multi-path approach and ensembling of wait-k models showed competitive latency-quality trade-offs, with ensembling boosting BLEU by 1.43 points on average.
- Speech-to-Text: The cascaded ASR+MT system, with a single ASR model and ensembled MT wait-k models, followed similar trends, with ensembling boosting BLEU by 0.67 points on average.
- ASR Performance: Offline ASR achieved a WER of 14.2% on MuST-C tst-COMMON, while the online ASR had a slightly higher WER of 16.3% (with an AL of 5s).
Limitations and Future Work
- Limitations:
- The paper notes that adding MuST-C synthetic data to certain pools surprisingly decreased BLEU scores on some test sets, and they couldn't fully investigate this due to time constraints.
- The impact of fine-tuning on specific models was found to be very limited.
- The performance of the cascaded ASR+MT system was still lower than offline systems, as expected due to the latency constraints.
- Future Work: The authors explicitly state that future work will be dedicated to simultaneous speech translation using end-to-end models, moving away from cascaded approaches.
Summary updated