TL;DR
This paper presents UPC's submission to IWSLT2023, tackling speech translation (ST) by leveraging large pre-trained speech (wav2vec 2.0) and text (mBART50) foundation models. They introduce an "Extended Siamese Pretraining" method that aligns speech and text encoder representations using Connectionist Temporal Classification (CTC) and Optimal Transport (OT) losses, significantly improving ST performance.
What Problem Does This Paper Address?
Speech Translation (ST) aims to directly translate spoken language into text in another language. While end-to-end models have advanced significantly, a key challenge is bridging the gap between speech and text representations, especially when using powerful pre-trained foundation models. These models, trained on different modalities, often have misaligned internal representations, hindering optimal transfer learning. This paper addresses this "representation discrepancy" by developing a pre-training strategy to align these representations before fine-tuning for the ST task.
Methods
The core of their approach is an Extended Siamese Pretraining strategy. Here's a breakdown:
- Foundation Models: They start with a wav2vec 2.0 model for speech (ASR foundation) and mBART50 for text (MT foundation).
- Siamese Pretraining: The key idea is to train the speech encoder and text encoder in parallel (Siamese) to make their output representations similar. This is achieved by:
- CTC Supervision: Applying CTC loss to the speech encoder's output to ensure good acoustic modeling.
- Optimal Transport (OT): Using OT (specifically Wasserstein distance) to align the speech encoder's representations with the text encoder's representations. They employ two OT losses: one for the input to the semantic encoder and another auxiliary loss for the semantic encoder's output.
- CTC Compression & Coupling Modules: To address the inherent length mismatch between speech and text, they use CTC-based compression to reduce the speech sequence length and introduce coupling modules (adapters and 1D convolutions) to process these compressed representations.
- Frozen Text Encoder: To preserve the rich semantic space of the pre-trained MT model, they freeze the text encoder during this pretraining phase.
- End-to-End Fine-tuning: After pretraining, the aligned speech encoder and the original MT decoder are fine-tuned end-to-end on ST data using Cross-Entropy loss. They also incorporate Knowledge Distillation (KD) from the MT model to further boost performance.
- Data Augmentation: They create synthetic data using SegAugment (segmenting existing data differently) and MT-based translation of existing transcripts to improve model robustness, especially for custom segmentations in test sets.
Datasets
- Training Data:
- ASR: MuST-C v3, Europarl-ST, CommonVoice v11.
- MT: Europarl v10 (for MT fine-tuning).
- ST: MuST-C v3, Europarl-ST, CoVoST-2.
- Synthetic Data: Generated using SegAugment on MuST-C v3 and MT translation of CommonVoice transcripts.
- Development/Test Data: IWSLT test sets (2019, 2020, ACLdev2023), MuST-C tst-COMMON.
Key Insights and Ideas
- Siamese Pretraining is Key: Aligning speech and text encoder representations before ST fine-tuning is crucial for maximizing the benefit of large foundation models. Simply using them separately doesn't fully exploit their potential.
- Optimal Transport for Alignment: OT proves to be an effective tool for quantitatively minimizing the discrepancy between speech and text representations, going beyond simple cross-modal attention.
- CTC for Acoustic Modeling & Compression: CTC serves a dual purpose: ensuring good acoustic modeling and providing a mechanism to compress speech sequences, helping to bridge the length gap with text.
- Synthetic Data Matters: Data augmentation, particularly with SegAugment and MT-generated data, significantly boosts performance, especially on test sets with custom segmentations (like IWSLT).
- ACL Domain Specificity: The models perform better on TED-like data than on the ACL presentation data, suggesting the latter is more out-of-domain. This also influences optimal segmentation strategies.
Key Results
- Strong Performance: Their best single model achieved 31.2 BLEU on MuST-C tst-COMMON, 29.8 on IWSLT.tst2020, and 33.4 on the newly released IWSLT.ACLdev2023.
- Siamese Pretraining Gains: The Siamese pretraining approach led to substantial improvements over baseline methods that only used CTC pretraining.
- Data Augmentation Impact: Incorporating synthetic data (SegAugment and CommonVoice translations) consistently improved results across test sets.
- Segmentation Sensitivity: Optimal segmentation parameters (min/max segment length) were found to be crucial and varied between TED-like and ACL domains, with shorter segments generally preferred for the latter.
Limitations and Future Work
- ACL Domain Performance: The models showed a performance gap on the ACL presentation data compared to TED talks, indicating potential for further domain adaptation.
- Ensemble Gains: Ensembling multiple models provided only small increases, suggesting that the individual models were already highly optimized and similar.
- Future Work: The authors plan to investigate the zero-shot capabilities of optimal transport in the context of foundation models.
Summary updated