Pretrained Speech Encoders and Efficient Fine-tuning Methods for Speech Translation: UPC at IWSLT 2022

Ioannis Tsiamas, Gerard I. Gállego, Carlos Escolano, José A. R. Fonollosa, Marta R. Costa-jussàACL 2022

TL;DR

This paper presents UPC's approach to the IWSLT 2022 Speech Translation tasks, focusing on end-to-end models that leverage large pretrained speech and text models. They introduce efficient fine-tuning techniques, including adapter modules and knowledge distillation, achieving state-of-the-art results for English-German speech translation.

What Problem Does This Paper Address?

End-to-end (E2E) Speech Translation (ST) systems, which directly translate speech to text without an intermediate transcription, have gained popularity over traditional cascaded systems. While E2E models avoid error propagation and offer faster inference, they typically require more data to be competitive. This paper addresses the challenge of training competitive E2E ST systems with limited data by effectively utilizing large pretrained speech and text models and exploring efficient fine-tuning strategies. They also tackle the issue of unsegmented test sets in benchmarks like IWSLT, which require a separate audio segmentation step.

Methods

The UPC team built their E2E ST systems using a combination of pretrained models:

  • Speech Encoders: They experimented with two popular self-supervised learning (SSL) based speech encoders: wav2vec 2.0 and HuBERT. These models are pre-trained on massive amounts of unlabeled speech data and then fine-tuned for Automatic Speech Recognition (ASR).
  • Text Decoder: The decoder is initialized from mBART50, a pretrained multilingual sequence-to-sequence model.
  • Length Adaptor: A crucial component to bridge the modality gap, this module reduces the sequence length of the encoder's output to better match the decoder's expectations.
  • Efficient Fine-tuning (LNA): Instead of fully fine-tuning the entire massive model, they employ a LayerNorm and Attention (LNA) strategy that selectively trains only a small percentage of parameters (layer normalization, self-attention, and cross-attention layers). This significantly reduces computational and memory requirements.
  • Parallel Adapters: To further improve performance, they integrate "adapters" – small feed-forward networks – into the frozen layers (specifically, the feed-forward blocks of the Transformer) of both the encoder and decoder. This allows for parameter-efficient adaptation without retraining large portions of the model.
  • Knowledge Distillation (KD): They explore transferring knowledge from a pretrained Machine Translation (MT) model (mBART) to the ST model. This involves a combined loss function that includes standard cross-entropy and a KL divergence term between the teacher (MT) and student (ST) output distributions.
  • Audio Segmentation (SHAS): For unsegmented test sets, they utilize a method called SHAS (Speech Translation Segmentation), which learns manual segmentation from labeled data and uses a Transformer-based frame classifier and a probabilistic divide-and-conquer algorithm.

Datasets

The researchers trained their models on a combination of three speech translation datasets:

  • MuST-C v2: English-German (en-de), English-Japanese (en-ja), and English-Chinese (en-zh). This dataset is based on TED talks and is considered "in-domain."
  • Europarl-ST: English-German (en-de). Derived from European Parliament proceedings.
  • CoVoST-2: English-German (en-de). Derived from the Common Voice corpus.

For evaluation, they used the official IWSLT test sets for 2019, 2020, 2021, and 2022. They also used the dev and tst-COMMON splits of MuST-C for development and testing.

Key Insights and Ideas

  • HuBERT outperforms wav2vec 2.0: For their ST system, the HuBERT encoder consistently yielded better results than wav2vec 2.0.
  • Adapters provide flexibility: While adapters sometimes led to a slight drop in performance on in-domain data (MuST-C), they improved performance on the IWSLT test sets, suggesting they offer valuable flexibility for handling data with different segmentation.
  • Knowledge Distillation's nuanced impact: KD showed only minor improvements on manually segmented data, likely due to the small performance gap between their ST and MT models. This suggests KD is most beneficial when the MT teacher is significantly better than the student ST model.
  • SHAS significantly improves segmentation: The custom SHAS segmentation method provided substantial gains (4-5.5 BLEU) compared to the provided segmentations on the IWSLT test sets.
  • Ensembling is key for top performance: Combining multiple diverse models (using different encoders, adapters, and KD) led to the best overall results.

Key Results

  • English-German: Their best single model, using HuBERT with LNA and parallel adapters, achieved 29.42 BLEU on MuST-C tst-COMMON and 26.77 BLEU on IWSLT 2020 test.
  • Ensembled Models: By ensembling their top models, they further boosted performance to 30.83 BLEU on MuST-C tst-COMMON and 27.78 BLEU on IWSLT 2020 test.
  • English-Japanese: Achieved 15.85 BLEU on MuST-C tst-COMMON.
  • English-Chinese: Achieved 25.63 BLEU on MuST-C tst-COMMON.
  • Segmentation Improvement: The SHAS method improved BLEU scores by 4 to 5.5 points across experiments compared to the given segmentation.

Limitations and Future Work

  • Knowledge Distillation Nuances: The effectiveness of KD was limited by the small performance gap between their ST and MT models, and the segmentation mismatch between training and testing data for IWSLT.
  • Adapters and Overfitting: Adapters sometimes led to overfitting on in-domain data.
  • Future Work: The authors plan to explore more pretrained speech encoders and text decoders, investigate optimal ways to combine them, and delve deeper into efficient fine-tuning strategies. They also aim to better leverage MT teachers in scenarios with smaller MT-ST performance gaps. They also submitted a Speech-to-Speech system by combining their ST model with a pretrained German TTS model.

Summary updated

BibTeX
@inproceedings{tsiamas2022pretrained,
  author = {Ioannis Tsiamas and Gerard I. Gállego and Carlos Escolano and José A. R. Fonollosa and Marta R. Costa-jussà},
  title = {{Pretrained Speech Encoders and Efficient Fine-tuning Methods for Speech Translation: UPC at IWSLT 2022}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure