TL;DR
This paper presents AppTek and RWTH Aachen University's participation in the IWSLT 2020 evaluation, showcasing both cascaded and end-to-end neural speech translation systems. They achieve competitive results by leveraging pre-training, data augmentation, and novel architectures for both offline and simultaneous translation, demonstrating significant progress in direct speech translation.
What Problem Does This Paper Address?
Developing high-quality speech translation systems, especially for real-time, simultaneous translation, is a significant challenge. Traditional cascaded systems, which first transcribe speech to text and then translate, often suffer from error propagation. End-to-end direct speech translation (DST) aims to overcome this by directly translating speech to text in the target language, but historically, DST systems have lagged behind cascaded systems in translation quality due to data scarcity and model complexity. This paper addresses the challenge of achieving competitive translation quality with both cascaded and end-to-end DST approaches, particularly focusing on improving simultaneous translation performance and closing the gap between cascaded and direct methods.
Methods
The authors explore several key methodological advancements:
- Cascaded Systems: They combine strong hybrid Automatic Speech Recognition (ASR) systems (using hybrid LSTM/HMM models and attention-based models) with Transformer-based Neural Machine Translation (NMT). For ASR, they utilize pre-training and data augmentation techniques like SpectAugment. For NMT, they employ both LSTM and Transformer architectures, including large Transformer models.
- End-to-End Direct Speech Translation (DST): This is a major focus. They leverage pre-trained ASR encoder and NMT decoder components, incorporating an "adaptor" layer to bridge the gap between them. They also heavily rely on data augmentation, including synthetic data generated from ASR transcripts and text-to-speech synthesis.
- Simultaneous Translation: A novel architecture is introduced that dynamically decides when to feed more input and when to generate output. This system is based on offline LSTM models but incorporates an additional binary output trained to predict chunk boundaries in the incoming speech stream, allowing for more flexible translation decisions.
- Data Preprocessing: For NMT, they preprocess English text to resemble ASR output by lowercasing, removing punctuation, expanding abbreviations, and converting numbers/dates to their spoken forms. This helps the NMT system learn to handle these elements during translation.
- Data Filtering and Domain Adaptation: They employ sentence embedding similarity and k-Means clustering to filter large out-of-domain corpora, improving the quality and relevance of training data.
Datasets
The paper utilizes several datasets for training and evaluation:
- ASR Training: TED-LIUM, LibriSpeech, Mozilla Common Voice, IWSLT TED corpora, EuroParl, How2, MuST-C.
- NMT Training: TED, MuST-C, News Commentary, Europarl, WikiTitles, ParaCrawl, CommonCrawl, rapid, OpenSubtitles.
- DST Training: MuST-C, IWSLT TED, EuroParl corpora. They also generate synthetic DST data by translating ASR transcripts (SYNTH_TRANS) and synthesizing speech from MT parallel data (SYNTH_SPEECH).
- Evaluation: TED tst2015, MuST-C tst-HE, MuST-C tst-COMMON, TED tst2019, TED tst2020.
Key Insights and Ideas
- Pre-training is Crucial for DST: Leveraging pre-trained ASR encoders and NMT decoders, along with an adaptor layer, significantly boosts DST performance, demonstrating the value of transfer learning in low-resource DST scenarios.
- Data Augmentation Bridges the Gap: Generating synthetic data (both translated text and synthesized speech) is vital for improving DST performance, especially when supervised DST data is scarce.
- Simultaneous Translation Can Be High Quality with Dynamic Decisions: The novel architecture for simultaneous translation, which makes dynamic chunk boundary decisions, shows that it's possible to achieve good translation quality even at low latencies, challenging the traditional latency-quality trade-off.
- Cascaded vs. Direct: The Gap is Closing: The paper shows that their end-to-end DST systems are approaching the performance of their best cascaded systems, indicating rapid progress in the field.
- ASR Output Emulation Benefits NMT: Preprocessing text to mimic ASR output helps NMT systems learn to handle spoken language nuances like numbers and dates more effectively.
Key Results
- Offline Translation: Their best cascaded system achieved a BLEU score of 30.9% on MuST-C tst-COMMON. Their ensemble of direct DST models reached 28.0% on TED tst2015, matching AppTek's 2018 cascaded system results.
- Simultaneous Translation: With a latency of around 4 seconds, their simultaneous translation systems achieved BLEU scores only 4 points behind their strong cascaded offline NMT baseline.
- DST Improvements: Pre-training yielded significant improvements in DST, with gains of up to 2.9% BLEU and 4.3% TER on average across test sets.
- Data Filtering Impact: Filtering out-of-domain data improved system performance by 1.6% BLEU on the development set.
Limitations and Future Work
- Data Sparseness in DST: While data augmentation and pre-training help, data sparseness remains a challenge for end-to-end DST. The paper explores various strategies to mitigate this.
- Simultaneous Translation Latency vs. Quality: While significant progress is made, there's still an inherent compromise between latency and translation quality in simultaneous systems. The paper shows that dynamic chunking helps, but further optimization is possible.
- Domain Mismatch in Synthetic Data: The paper notes that synthetic data generated via translation (SYNTH_TRANS) can sometimes lead to slightly worse results, potentially due to domain mismatch. Fine-tuning on generated data helps mitigate this.
- Future Work: The authors suggest exploring further improvements in data augmentation, architecture design for both cascaded and direct systems, and more sophisticated methods for handling latency in simultaneous translation. They also highlight the continued progress in closing the gap between cascaded and direct DST approaches.
Summary updated