TL;DR
This paper presents an end-to-end speech translation system for the IWSLT 2021 task, leveraging pre-trained Wav2Vec 2.0 and mBART models. They introduce an "Adapter" module and a two-step fine-tuning strategy to improve performance and efficiency, achieving a strong BLEU score of 28.22 on the MuST-C test set.
What Problem Does This Paper Address?
Speech translation (ST) systems traditionally rely on a cascade of Automatic Speech Recognition (ASR) and Machine Translation (MT) models. While end-to-end (E2E) systems offer advantages like reduced error propagation and faster inference, they face challenges due to the scarcity of large ST datasets and the inherent difficulty of jointly performing acoustic and semantic modeling. This paper aims to push the boundaries of E2E ST by effectively utilizing pre-trained models and introducing novel architectural and training techniques to bridge the gap with cascade systems.
Methods
The core of the UPC system is an end-to-end encoder-decoder architecture. They utilize a pre-trained Wav2Vec 2.0 model as the encoder and a pre-trained mBART decoder. To facilitate the integration of these powerful but distinct pre-trained models, they introduce two key components:
- Adapter Module: Inspired by previous work, this module is inserted after the Wav2Vec 2.0 encoder to help adapt its output representations to the mBART decoder's modality. This is crucial for bridging the "multimodal gap."
- Two-Step Fine-tuning Strategy: Instead of fine-tuning all parameters at once, they propose a two-step approach. First, they train only the "coupling modules" (Adapter and Length Adaptor) while keeping the pre-trained encoder and decoder frozen. In the second step, they fine-tune the entire system. This strategy is hypothesized to prevent the initially random weights of the coupling modules from hindering the learning process and to accelerate convergence.
Additionally, they employ:
- Data Augmentation: Randomized on-the-fly augmentations like echo effects, tempo, and pitch modifications are applied to the source audio to improve robustness. Since their system works directly on audio waveforms, they mask the output of the Wav2Vec 2.0 feature extraction module as an alternative to SpecAugment.
- Custom Data Segmentation: Recognizing that the provided segmentation in IWSLT tasks is often not sentence-like, they develop a custom segmentation algorithm. This algorithm uses a pre-trained Wav2Vec 2.0 model to identify untranscribable speech segments, aiming to create more natural sentence boundaries.
Datasets
The experiments utilize several datasets:
- Training Data: A concatenation of the English-to-German splits from MuST-C v2, EuroparlST, and CoVoST 2. The EuroparlST "train-noisy" split was excluded due to quality concerns.
- Development Data: The development split of MuST-C, augmented with the development splits of EuroparlST and CoVOST.
- Test Data:
- MuST-C test set for in-domain evaluation.
- IWSLT 2019 test set for evaluating the custom segmentation algorithm.
- IWSLT 2020 and 2021 test sets for tracking year-to-year progress (references for these were not publicly available at the time of submission).
Key Insights and Ideas
- Pre-trained models are powerful for ST: The paper reinforces the idea that leveraging large pre-trained models like Wav2Vec 2.0 and mBART is highly effective for speech translation, even in bilingual scenarios.
- Adapters bridge modality gaps: The Adapter module proves beneficial in adapting the encoder's output to the decoder, leading to improved translation quality.
- Two-step fine-tuning accelerates and improves: The proposed two-step fine-tuning strategy, where coupling modules are trained first, not only speeds up training but also leads to better final performance by providing a more stable initialization.
- Custom segmentation is crucial: For tasks like IWSLT where provided segmentations are suboptimal, developing a custom segmentation algorithm that leverages ASR models can significantly boost performance.
- Minimal fine-tuning is effective: The paper confirms the findings of Li et al. (2021) that training only a fraction of parameters (around 20%) can achieve performance comparable to full fine-tuning, making training large models more feasible.
Key Results
- The baseline end-to-end system (LNA-ED) achieved a BLEU score of 26.23 on the MuST-C test set.
- Adding the Adapter module (LNA-ED-Adapt) improved the BLEU score to 26.99 on MuST-C.
- The two-step fine-tuning strategy (LNA-ED-Adapt-2step) further boosted performance to 27.25 BLEU on MuST-C.
- Fine-tuning the best single model on in-domain data yielded a score of 27.25 BLEU on MuST-C.
- An ensemble of the two best single models achieved the top score of 28.22 BLEU on the MuST-C test set.
- The custom segmentation algorithm provided improvements of 2.5 to 3 BLEU points on the IWSLT 2019 test set compared to using the given segmentation.
- The ensemble model achieved a BLEU score of 18.3 on the IWSLT 2021 test set against TED references and 21.8 against IWSLT references.
Limitations and Future Work
- Data Scarcity: The paper acknowledges the general challenge of limited data for ST tasks, which is why pre-trained models are essential.
- Computational Cost: While the minimal fine-tuning strategy helps, training large models still requires significant computational resources.
- Future Work: The authors suggest that future research should focus on exploring better methods to adapt pre-trained models to new languages and tasks, with Adapter modules being promising candidates. They also highlight the effectiveness of their approach for bilingual scenarios and its potential for broader application.
Summary updated