Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23

Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, Marta R. Costa-jussà

TL;DR

The authors present the Universitat Politècnica de Catalunya (UPC) submission for the IWSLT 2023 Offline Speech Translation task, focusing on English-to-German translation. By introducing an "Extended Siamese pretraining" method that leverages Connectionist Temporal Classification (CTC) and Optimal Transport (OT), they successfully align the representation spaces of speech (wav2vec 2.0) and text (mBART50) foundation models. This approach significantly mitigates modality discrepancies and maximizes transfer learning from machine translation, yielding state-of-the-art end-to-end translation performance.

What Problem Does This Paper Address?

End-to-end Speech Translation (ST) models directly translate audio into target text, bypassing the error-propagation issues common in traditional cascaded systems (ASR + MT). While leveraging large, self-supervised foundation models for speech and text seems like an obvious win, it introduces a major hurdle: representation discrepancy.

Speech features and text tokens reside in entirely different semantic spaces and possess vastly different sequence lengths. Simply plugging a speech encoder into a text decoder often fails because the decoder doesn't understand the encoder's unaligned representations. The authors address this gap by forcing the speech encoder to map its outputs directly into the text model's pre-existing semantic space.

Methods

The authors build upon recent Siamese pretraining techniques by introducing an architecture that decouples acoustic and semantic modeling:

  1. Dual-Encoder Architecture: The speech encoder consists of an acoustic encoder (initialized from wav2vec 2.0 fine-tuned on ASR) and a semantic encoder (initialized from mBART50's text encoder).
  2. CTC Compression & Coupling: To resolve length mismatches, a CTC module compresses the acoustic representations by averaging repeating predictions and removing blank tokens. A subsequent strided 1D convolution further downsamples the sequence to transition from character-level to sub-word-level lengths.
  3. Extended Siamese Pretraining: The speech encoder is trained to mimic a frozen text encoder. Crucially, the losses are decoupled: a CTC loss guides the acoustic encoder, while two separate Optimal Transport (OT) losses align the inputs and outputs of the semantic encoder with the text space.
  4. End-to-End Fine-Tuning: After alignment, the acoustic parts are frozen, and the entire system is fine-tuned end-to-end with the mBART50 decoder using Cross-Entropy loss and Knowledge Distillation (KD) from the original MT model.

Extended Siamese pretraining framework Speech Translation fine-tuning stage

Datasets

The models were trained and evaluated using several prominent corpora:

  • Speech Translation (ST): MuST-C v3 (TED talks), Europarl-ST (European Parliament), and CoVoST-2.
  • ASR/Pretraining: English ASR data extracted from MuST-C v3, Europarl-ST, and CommonVoice v11.
  • Data Augmentation: The authors generated synthetic ST data by using the MT foundation model to translate CommonVoice v11 transcripts. They also utilized SegAugment to create alternative audio segmentations of MuST-C v3.
  • Evaluation Benchmarks: IWSLT test sets from 2019 and 2020 (TED domain), alongside the newly released IWSLT.ACLdev2023 (academic talk domain).
Filtered training data statistics

Key Insights and Ideas

  • Decoupling Losses is Crucial: Applying CTC strictly to acoustic representations and OT strictly to semantic representations allows each encoder segment to specialize, leading to cleaner cross-modal alignment.
  • Preserving the MT Space: Freezing the text encoder during the Siamese pretraining phase ensures that the well-structured machine translation space remains intact, preventing representation drift.
  • Domain-Specific Audio Segmentation: Optimal audio segment lengths vary dramatically by domain. While standard TED talks benefit from longer segments (10–26 seconds), out-of-domain academic presentations (ACL talks) perform significantly better with very short segments (0.2–12 seconds). The authors hypothesize that when models encounter out-of-domain audio, their confidence drops, making shorter segments easier to process accurately.

Key Results

The Extended Siamese pretraining strategy demonstrated clear advantages across all evaluation sets:

  • Baseline vs. Siamese: Introducing Siamese pretraining boosted performance on the out-of-domain acl2023 set from 25.1 to 26.4 BLEU using only MuST-C data.
  • Data Scaling: Incorporating diverse datasets (CommonVoice and CoVoST-2) dramatically enhanced generalization, driving the acl2023 score up to 31.6 BLEU due to better representation of non-native accents.
  • Top Performers: The best single model achieved 31.2 BLEU on MuST-C v3, 29.8 BLEU on tst2020, and 33.4 BLEU on ACLdev2023.
BLEU score results across configurations

Limitations and Future Work

  • Low Ensemble Diversity: Ensembling multiple top-performing models provided only marginal gains. Because the models were all initialized from the same Siamese pretraining checkpoint, the final fine-tuned iterations lacked the diversity required for a highly effective ensemble.
  • Future Directions: The authors plan to investigate the zero-shot capabilities of optimal transport within foundation models to see if cross-modal alignment can be achieved even more efficiently without heavy pretraining phases.

Summary updated

Abstract

This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023.
BibTeX
@inproceedings{tsiamas2023speech,
  author = {Ioannis Tsiamas and Gerard I. Gállego and José A. R. Fonollosa and Marta R. Costa-jussà},
  title = {{Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure