End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021

Gerard I. Gállego, Ioannis Tsiamas, Carlos Escolano, José A. R. Fonollosa, Marta R. Costa-jussà

TL;DR

This paper presents an efficient end-to-end speech translation system that successfully couples pre-trained speech (Wav2Vec 2.0) and text (mBART) models. By introducing a lightweight Adapter module and a smart two-step training strategy, the authors achieve state-of-the-art translation quality while fine-tuning only 20% of the model's 770 million parameters.

What Problem Does This Paper Address?

Traditional speech translation (ST) relies on "cascade" systems that first transcribe audio to text using Automatic Speech Recognition (ASR) and then translate it using Machine Translation (MT). While effective, cascades suffer from error propagation (if the ASR messes up, the translation fails) and high latency.

End-to-end (E2E) speech translation directly maps audio to target text, solving these issues. However, E2E systems are notoriously hard to train due to data scarcity and the "multimodal gap"—the inherent difficulty of aligning continuous acoustic waveforms with discrete semantic text tokens. This paper tackles how to efficiently stitch together powerful, disparate pre-trained models (one for audio, one for text) to create a high-performing E2E translation system without requiring massive computational resources to train from scratch.

Methods

The authors build an E2E architecture by combining two heavyweights in self-supervised learning: Wav2Vec 2.0 (as the audio encoder) and mBART50 (as the text decoder). To make them work together seamlessly, they introduce two coupling components:

  1. The Adapter Module: Placed right after the encoder, this module bridges the modality gap. It projects the audio representations into a higher-dimensional space, applies a ReLU activation and LayerNormalization, and uses a residual connection to refine the features before passing them to the decoder.
  2. The Length Adaptor: Audio signals are much longer than text sequences. This module uses a stack of 1D convolutional layers to down-sample the encoder's output by 8x, matching the expected sequence length of the decoder.
Figure 1: System overview. The original architecture proposed by Li et al. (2021) includes a pre-trained Wav2Vec 2.0 as the encoder, a pre-trained mBART de- coder and a Length Adaptor. In this work, we add an Adapter module after the encoder.

Efficient Fine-Tuning & Two-Step Training

Instead of updating all 770 million parameters, the authors adopt the LNA (LayerNorm and Attention) fine-tuning strategy. They freeze the bulk of the pre-trained weights and only train the LayerNormalization layers, encoder self-attention, and encoder-decoder cross-attention—amounting to just 20% of the total parameters.

Crucially, they propose a two-step training strategy (LNA-ED-Adapt-2step). Because the coupling modules are initially randomized, training the whole system immediately can corrupt the fragile pre-trained representations. They first freeze everything except the coupling modules for 4 epochs, allowing them to stabilize, before opening up the rest of the LNA parameters.

Additionally, they implement a custom audio segmentation algorithm driven by Wav2Vec 2.0 to detect untranscribable periods (like silence or background noise) and split long audio files into optimal, sentence-like chunks.

Datasets

The system was trained and evaluated on English-to-German data across three major speech translation datasets:

  • MuST-C v2: Used as the primary in-domain dataset (derived from TED talks).
  • EuroparlST: Out-of-domain dataset consisting of parliamentary debates.
  • CoVoST 2: Out-of-domain massively multilingual speech-to-text dataset.

Key Insights and Ideas

  • Adapters Smooth the Multimodal Transition: Simply feeding audio features into a text decoder creates friction. The tiny Adapter module (which only added 5% to the total trainable parameters) acts as a crucial translator between the acoustic and textual latent spaces.
  • Don't Let Random Weights Bully Pre-trained Models: The two-step training approach proved vital. Letting the coupling modules "warm up" first prevented random gradients from disrupting the highly optimized, pre-trained weights of Wav2Vec 2.0 and mBART, leading to faster convergence.
Figure 4: BLEU scores on MuST-C-dev during training
  • Smart Segmentation is an Easy Win: Standard automatic test-set segmentations are often poorly aligned with actual sentences. By using an ASR-based custom segmentation method, the authors unlocked massive performance gains without changing the core translation model architecture.

Key Results

  • Baseline vs. Adapter: Adding the Adapter module increased the MuST-C test score from 26.23 to 26.99 BLEU.
  • Two-Step Success: The two-step training method further pushed the single-model score to 27.25 BLEU.
  • Final Ensemble: Ensembling their two best variations yielded a final score of 28.22 BLEU on the MuST-C test set.
  • The Power of Segmentation: On the IWSLT 2019 test set, switching from the provided segmentation to their custom Wav2Vec 2.0-based segmentation boosted performance by 2.5 to 3 BLEU points across the board.
  • Competitive Standings: Evaluated on the IWSLT 2020 test set, their system achieved 24.6 BLEU, which would have placed 3rd on the previous year's highly competitive leaderboard—just 0.7 BLEU behind the top heavily engineered system.

Limitations and Future Work

While the parameter-efficient LNA strategy works remarkably well, the authors note a risk of catastrophic forgetting when further fine-tuning the model strictly on narrow, in-domain data (like MuST-C). The model can lose some of its generalized translation capabilities.

The authors suggest that future work should focus on discovering even better methods to adapt these massive pre-trained models to new languages and downstream tasks, highlighting specialized Adapter modules as the most promising path forward.

Summary updated

Abstract

This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuST-C test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation.
BibTeX
@inproceedings{gallego2021end,
  author = {Gerard I. Gállego and Ioannis Tsiamas and Carlos Escolano and José A. R. Fonollosa and Marta R. Costa-jussà},
  title = {{End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure