TL;DR
The USTC-NELSLIP team presents their robust systems for the IWSLT 2023 Offline Speech Translation task, focusing on English-to-Chinese translation. They achieve strong results by combining a pre-trained Whisper ASR model with advanced MT techniques in a cascaded system, and by proposing a novel end-to-end architecture called SATE-ex that effectively fuses acoustic and textual information.
What Problem Does This Paper Address?
Speech Translation (ST) aims to directly translate spoken language into written text in another language. This is a complex task that bridges Automatic Speech Recognition (ASR) and Machine Translation (MT). Traditional approaches often use a cascaded system: first, ASR transcribes the speech into text, and then MT translates this text. However, errors in the ASR step can propagate and significantly degrade the final translation quality. End-to-end (E2E) systems aim to overcome this by directly translating speech to text, potentially reducing error propagation but often facing challenges in achieving the same performance as well-tuned cascaded systems. The IWSLT 2023 Offline Speech Translation task specifically challenges systems to perform this translation without relying on internet connectivity, demanding efficient and robust models. The USTC team tackles these challenges by improving both cascaded and E2E approaches, focusing on error reduction and better information fusion.
Methods
The USTC team employed two main strategies:
- Cascaded Systems:
- ASR Enhancement: They integrated the pre-trained Whisper model (Radford et al., 2022) as their ASR component. Whisper, trained on massive amounts of weakly supervised data, significantly reduces errors in the intermediate text transcription compared to previous methods.
- MT Techniques: For the machine translation part, they utilized a suite of advanced techniques including Back Translation, Sentence-level Knowledge Distillation, Domain Adaptation, and Robust MT Training. These methods help improve translation accuracy and robustness against potential ASR errors.
- Ensembling: Multiple ASR and MT models were trained and ensembled to further boost performance.
- End-to-End (E2E) Systems:
- SATE-ex (Stacked Acoustic-and-Textual Encoding extension): This is their novel E2E architecture, building upon the SATE (Stacked Acoustic-and-Textual Encoding) framework. SATE-ex features an acoustic encoder and a textual encoder. A key innovation is the Cross-Attention module within the textual decoder, which allows it to directly process the output of the acoustic decoder. This facilitates the fusion of acoustic and textual information, aiming to improve representation consistency and mitigate error propagation.
- Initialization: Both acoustic and textual encoders/decoders in SATE-ex are initialized using components from pre-trained ASR and MT models, leveraging existing knowledge.
- Adaptation Training: To bridge the gap between pre-training and fine-tuning, they employed adaptation training, including multi-teacher knowledge distillation.
- Ensembling: Similar to the cascaded approach, they ensembled multiple SATE-ex and encoder-decoder based E2E models.
Datasets
The team utilized a variety of datasets for training and evaluation:
- ASR Training: Librispeech, Europarl, MuST-C (v1, v2), TED LIUM3, VoxPopuli, and CoVoST2. They also employed extensive data augmentation techniques, including speed perturbation, oversampling, and generating synthetic audio using Glow-TTS on text corpora, resulting in a substantial amount of training data (up to 56,000 hours).
- MT Training: CCMT Corpus, News Commentary, ParaCrawl, Wiki Titles, UN Parallel Corpus, WikiMatrix, MuST-C, and CoVoST2 for parallel data. They also used News crawl and Common Crawl for monolingual data to facilitate back-translation.
- Speech Translation Evaluation: MuST-C (v2) tst-COMMON and Development sets, and the IWSLT 2022 test sets (tst2018, tst2019, tst2020, tst2022). They also created "KD Corpus" and "TTS Corpus" for data augmentation.
Key Insights and Ideas
- Whisper's Power: The pre-trained Whisper model proved to be a highly effective ASR component, significantly reducing transcription errors and boosting cascaded system performance. Its ability to transfer well to new datasets without extensive fine-tuning is a major advantage.
- SATE-ex: Bridging Modalities: The SATE-ex architecture's core idea of using cross-attention to directly fuse acoustic decoder outputs into the textual decoder is crucial. This allows for a more direct and potentially more effective integration of speech and text information compared to simply passing text.
- Data Augmentation is Key: The extensive use of data augmentation, especially generating synthetic speech (TTS) and leveraging large-scale pre-trained models, was vital for improving both ASR and ST performance, particularly for the E2E systems.
- Ensembling for Robustness: Combining multiple models, whether cascaded or E2E, consistently improved performance, highlighting the benefit of diversity in model training.
Key Results
- ASR Performance: The pre-trained Whisper model achieved superior recognition accuracy compared to their own ASR fusion model, with an average improvement of 0.51% across various test sets.
- Cascaded Systems: The cascaded system using Whisper as ASR and an ensembled MT model (System #7) achieved strong results, outperforming their ASR+MT system (System #6) on development sets.
- End-to-End Systems: Ensemble SATE-ex models (System #16) outperformed ensemble encoder-decoder models (System #17), demonstrating the effectiveness of their proposed architecture.
- Overall Performance: The paper reports BLEU scores across various systems and test sets, with their best cascaded and E2E ensembles achieving competitive results, particularly System #18 (ensemble of cascaded and E2E) and System #20 (contrastive ensemble).
Limitations and Future Work
- SATE-ex Fine-tuning: The paper notes that a single SATE-ex model performed slightly worse than a single encoder-decoder model, attributing this to a lack of fine-tuning for the SATE-ex architecture. This suggests potential for further improvement with more targeted fine-tuning.
- Future Directions: The authors express interest in further investigating SATE-ex and exploring multimodal representation learning in speech translation. This indicates a direction towards more deeply integrating different modalities beyond just acoustic and textual features.
Summary updated