Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition

Bethan Thomas, Samuel Kessler, Salah KaroutICASSP 2022 · DOI

TL;DR

This paper introduces a more efficient way to adapt large, pre-trained self-supervised speech models (like wav2vec 2.0) for Automatic Speech Recognition (ASR). By using "adapters" – small, trainable modules inserted into the model – they achieve comparable ASR performance to full fine-tuning while training only a fraction of the parameters, making it significantly more scalable and accessible.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has revolutionized speech processing, enabling models to learn powerful representations from vast amounts of unlabeled audio data. State-of-the-art models like wav2vec 2.0 and HuBERT are pre-trained this way and then fine-tuned on smaller, labeled datasets for downstream tasks like ASR. However, this fine-tuning process is computationally expensive, requiring the retraining and storage of a large portion of the model's parameters for each new task or language. This poses a significant challenge for scaling to multi-lingual ASR or adapting models to numerous tasks, as it necessitates storing potentially millions of parameters per task. The paper aims to address this by proposing a parameter-efficient transfer learning method.

Methods

The core of this paper's contribution lies in applying "adapter modules" to self-supervised speech models for ASR. Adapters are small, lightweight neural network modules that are inserted into the layers of a pre-trained, frozen network. During fine-tuning, only the adapter modules, along with a few other components like layer normalization and a final linear classifier, are trained. The rest of the pre-trained model remains fixed.

The authors specifically insert adapter modules twice within each transformer block of the wav2vec 2.0 architecture. They also explore placing adapters in different layers, finding that concentrating them in the top few layers yields performance close to full fine-tuning. The adapters themselves are designed with a down-projection, a non-linearity, and an up-projection, with a skip connection to allow them to be bypassed if not needed. This approach contrasts with traditional fine-tuning, where a large percentage of the model's parameters are updated.

Datasets

  • LibriLight (LL): A 10-hour supervised subset was used for English ASR experiments.
  • LibriSpeech: The dev-clean and dev-other sets were used for evaluating English ASR performance.
  • Common Voice (CV): A 10-hour subset of the French corpus was used for French ASR experiments and evaluation on the CV test set.
  • Bi-lingual (French and English) wav2vec 2.0 model: This model was pre-trained on approximately 1000 hours of French CV data and 960 hours of English LibriSpeech data for multi-lingual experiments.

Key Insights and Ideas

  • Adapters offer a significant parameter efficiency boost: The most striking insight is the dramatic reduction in trainable parameters (down to ~9.2% compared to ~95.6% for full fine-tuning) without a substantial drop in ASR performance. This directly addresses the scalability issue.
  • Top layers are crucial for ASR: The ablation studies confirm the hypothesis that higher layers in self-supervised speech models encode more phonemic information, making them more critical for downstream ASR tasks. Adapters in these top layers are sufficient for good performance.
  • Adapters are more robust to language mismatch: When transferring a model to a different language (e.g., French), adapters show a slight performance increase compared to fine-tuning. This is attributed to adapters preserving the original pre-trained network's knowledge, allowing them to compensate for language differences more effectively than fine-tuning, which overwrites this information.
  • Adapters simplify the training process: Adapter experiments are faster and require fewer training steps, making them more experimentally friendly.

Key Results

  • English ASR: Adapters achieved performance very close to full fine-tuning on LibriSpeech dev-clean (9.39% WER with adapters vs. 8.98% WER with fine-tuning) while training only 9.2% of parameters.
  • French ASR: Adapters achieved comparable performance to fine-tuning (39.4% WER with adapters vs. 40.2% WER with fine-tuning) on the Common Voice test set.
  • Layer Ablations: Using adapters in just the top 4 layers of the 12-layer wav2vec 2.0 model yielded nearly as good performance as using adapters in all layers. Optimizing fine-tuning required training the top 8 layers.
  • Parameter Scalability: Figure 3 visually demonstrates how fine-tuning tasks nearly double the required parameters, while adapters add only a small, constant overhead, highlighting their superior scalability for multi-task scenarios.

Limitations and Future Work

  • Performance Gap: While adapters achieve close performance, there is still a slight gap compared to full fine-tuning in some English ASR scenarios.
  • Future Work: The authors suggest exploring adapters for other downstream tasks like speaker recognition and speech translation. They also propose investigating layer ablations in the NLP domain, drawing parallels between the two fields. The paper also hints at the potential for adapters to benefit from future advancements in NLP self-supervised learning.

Summary updated

BibTeX
@inproceedings{thomas2022efficient,
  author = {Bethan Thomas and Samuel Kessler and Salah Karout},
  title = {{Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746223},
}
Figure