TL;DR
The USTC-NELSLIP team presents their systems for the IWSLT 2022 Offline Speech Translation task, showcasing improvements in both cascaded (ASR + MT) and end-to-end approaches. They achieved significant gains by leveraging robust training techniques, data augmentation, and model ensembling, demonstrating the continued competitiveness of cascaded systems while pushing the boundaries of end-to-end performance.
What Problem Does This Paper Address?
Speech Translation (ST) aims to convert spoken language from one language to another. Traditionally, this has been tackled using a cascaded approach: first, an Automatic Speech Recognition (ASR) system transcribes the speech into text, and then a Machine Translation (MT) system translates this text. While effective, this pipeline can suffer from error propagation, where errors in the ASR stage are amplified by the MT system. More recently, end-to-end (E2E) models have emerged, aiming to directly translate speech to text in a single neural network, promising to simplify the architecture and potentially mitigate error propagation. However, E2E models have historically lagged behind cascaded systems in performance. This paper addresses the challenge of improving both cascaded and E2E ST systems, aiming to close the performance gap and explore techniques for robust and efficient speech translation.
Methods
The USTC-NELSLIP team employed a combination of established and novel techniques for both cascaded and end-to-end ST:
- Cascaded Systems:
- ASR: They experimented with VGG-Transformer, VGG-Conformer, and GateCNN-Conformer architectures, incorporating techniques like SpecAugment for robustness and Connectionist Temporal Classification (CTC) for better convergence.
- MT: Transformer-based models were used, enhanced by:
- Back Translation: Generating synthetic parallel data by translating monolingual target data into the source language.
- Knowledge Distillation (Self-Training): Using a trained MT model to generate translations for English sentences, which are then used to augment training data.
- Domain Adaptation: Fine-tuning models on in-domain data (e.g., MuST-C, TED-LIUM3) to improve performance on specific datasets.
- Denoising: A technique to measure and select data for MT training, aiming to reduce the negative impact of noisy data.
- Ensembling: Combining multiple Transformer variants to achieve better performance.
- Robust MT Training: A key innovation is an ASR output adaptation method to improve MT robustness against ASR errors. This involves fine-tuning the MT model with synthetic data generated by a trained ASR model, using KL loss to prevent overfitting, and distilling the model with both clean and ASR-outputted inputs.
- Segmentation: They utilized Supervised Hybrid Audio Segmentation (SHAS) for segmenting audio, and also explored a "semantic SHAS" approach.
- End-to-End Systems:
- Model Architectures: They investigated four variants based on VGG-Conformer and VGG-Transformer encoders, with Transformer decoders. Some models were initialized with pre-trained ASR components (e.g., VGG-Conformer from ASR), while others were initialized with pre-trained MT components.
- Data Augmentation: Similar to cascaded systems, they leveraged TTS-generated audio features and knowledge distillation to expand training data.
- Ensembling: Multiple E2E models were ensembled to boost performance.
Datasets
The team utilized a variety of datasets for training and evaluation across different tasks:
- ASR Training: Librispeech, Europarl, MuST-C(v1, v2), TED-LIUM3, CoVoST2, and VoxPopuli. They also augmented these with concatenated adjacent voices and TTS-generated audio features, resulting in a substantial amount of training data (up to 56,000 hours).
- Text MT Training: For EN→DE, EN→ZH, and EN→JA, they used a combination of parallel corpora (e.g., CommonCrawl, CoVoST2, Europarl, MuST-C) and monolingual data for back-translation.
- Speech Translation Training: MuST-C, CoVoST2, and Europarl were used. They further augmented these with "KD Corpus" (generated via text translation) and "TTS Corpus" (generated using Glow-TTS), significantly increasing the training data size.
- Evaluation: The primary evaluation was performed on IWSLT 2022 test sets, with comparisons also made to previous IWSLT test sets (tst-COM, tst2018, tst2019, tst2020).
Key Insights and Ideas
- Robustness is Key: The proposed robust MT training method, which explicitly trains the MT model to handle ASR errors, yielded significant improvements in cascaded ST performance. This highlights the importance of addressing the ASR-MT interface directly.
- Data Augmentation Pays Off: Both TTS-generated audio and knowledge distillation significantly boosted performance for both ASR and E2E ST systems, underscoring the value of synthetic data.
- Cascaded Systems Still Reign Supreme (for now): Despite advancements in E2E models, the paper consistently shows that their cascaded systems achieve higher BLEU scores, indicating that the pipeline approach, when optimized, remains the state-of-the-art for this task.
- Initialization Matters for E2E: Initializing E2E models with pre-trained ASR or MT components showed varying degrees of success, suggesting that leveraging existing knowledge is beneficial.
- Ensembling is a Reliable Booster: Ensembling multiple models, whether ASR, MT, or E2E, consistently led to improved performance across the board.
Key Results
- Cascaded ST: Achieved a 2.72 BLEU improvement over their previous year's system on the MuST-C English-German test set.
- End-to-End ST: Showed a 2.26 BLEU improvement over their previous year's E2E system.
- EN→DE: Their best cascaded system achieved 34.47 BLEU on tst-COM, outperforming their E2E system by 1.73 BLEU on average.
- EN→ZH: Their best cascaded system achieved 29.80 BLEU on tst-COM, outperforming their E2E system by 0.89 BLEU.
- EN→JA: Their best cascaded system achieved 22.02 BLEU on tst-COM, outperforming their E2E system by 2.89 BLEU.
- ASR: The GateCNN-Conformer with TTS augmented data performed best as a single model, with an ensemble achieving a WER of 5.32.
Limitations and Future Work
- E2E vs. Cascaded Gap: While E2E systems showed improvement, they still lag behind cascaded systems. The authors suggest investigating speech data augmentation and multi-modal representations for future E2E work.
- Semantic SHAS: While semantic SHAS was explored, it showed slightly worse performance than standard SHAS in some scenarios, indicating room for further refinement.
- Punctuation in EN→JA: The lack of punctuation in Japanese test sets and references impacted performance, suggesting that handling punctuation more effectively could be a future direction.
Summary updated