PARAMETER-EFFICIENT TRANSFER LEARNING OF AUDIO SPECTROGRAM TRANSFORMERS

Umberto Cappellazzo, Daniele Falavigna, Alessio Brutti, Mirco RavanelliMLSP 2024 · DOI · OpenReview

TL;DR

This paper tackles the challenge of efficiently adapting the powerful Audio Spectrogram Transformer (AST) model to various audio and speech tasks using parameter-efficient transfer learning (PETL). They introduce a novel "Conformer adapter" that leverages depthwise convolutions, achieving state-of-the-art performance with a tiny fraction of trainable parameters, even outperforming full fine-tuning on some tasks.

What Problem Does This Paper Address?

Large pre-trained models have revolutionized fields like NLP and audio processing. The standard approach to adapt these models to new tasks is "full fine-tuning," where all model parameters are updated. While effective, this creates a unique, large model for every downstream task, which is computationally expensive and impractical for deployment on resource-constrained devices or when dealing with many tasks. Parameter-efficient transfer learning (PETL) offers a solution by freezing most of the pre-trained model and training only a small number of additional parameters per task. However, despite the success of AST in audio classification, there was a surprising lack of research on efficiently adapting it using PETL methods. This paper aims to fill that gap by investigating existing PETL techniques and proposing a new, more effective adapter for AST.

Methods

The authors first conduct a comprehensive study of common PETL methods, including:

  • LoRA (Low-Rank Adaptation): This method injects trainable low-rank matrices into the pre-trained weight matrices to approximate weight updates.
  • Prompt/Prefix-Tuning: These techniques involve adding trainable continuous embeddings (prompts) to the input or hidden states of the transformer layers.
  • Adapter-Tuning: This involves inserting small, trainable "adapter" modules within the transformer layers. The paper specifically examines the "bottleneck adapter," which uses a low-dimensional bottleneck to reduce parameter count.

The core novelty of this paper is the Conformer Adapter. Inspired by the Conformer model's success in speech processing, this adapter incorporates a convolution module, specifically a depthwise convolution layer. This design choice is motivated by the ability of convolutions to capture local spatial correlations in spectrograms, which is crucial for audio tasks, and their inherent parameter efficiency compared to standard convolutions. The Conformer adapter down-projects the input, uses a Gated Linear Unit (GLU), applies the depthwise convolution, followed by Batch Normalization and Swish activation, and finally up-projects the output.

Datasets

The study evaluates PETL methods on four audio/speech downstream classification tasks:

  • Audio Classification:
    • ESC-50: 2,000 5-second environmental audio recordings across 50 classes.
  • UrbanSound8K (US8K): 8,732 labeled urban sound excerpts from 10 classes.
  • Keyword Spotting:
    • Speech Commands V2 (GSC): 105,829 1-second recordings of 35 speech commands.
  • Intent Classification:
    • Fluent Speech Commands (FSC): 30,043 English utterances spanning 31 classes.
  • Emotion Recognition:
    • IEMOCAP (IEM): 10,039 utterances from 10 speakers with 4 emotional classes.

The AST model used was pre-trained on ImageNet-21K and AudioSet. For additional experiments, the Wav2Vec 2.0 model was also utilized.

Key Insights and Ideas

  • The limitations of simple bottleneck adapters: The paper suggests that the purely linear design of traditional bottleneck adapters might not be sufficient to capture the complex spatial relationships in audio spectrograms, leading to performance gaps on speech tasks.
  • Depthwise convolutions are key for audio: The success of the Conformer adapter highlights the effectiveness of depthwise convolutions in capturing local spatial features in audio spectrograms while remaining parameter-efficient.
  • Conformer adapter excels in speech tasks: The proposed adapter shows significant improvements, especially on speech-centric tasks like GSC and FSC, where it can even surpass full fine-tuning.
  • Parameter efficiency doesn't always mean sacrificing performance: The Conformer adapter demonstrates that it's possible to achieve state-of-the-art results with a minuscule percentage of trainable parameters (as low as 0.29%).
  • Kernel size matters: The choice of kernel size for the depthwise convolution in the Conformer adapter is shown to be important, with larger kernels being more beneficial for challenging datasets like GSC, likely due to their ability to learn more global features.

Key Results

  • The proposed Conformer adapter consistently outperforms other PETL methods, including LoRA and bottleneck adapters, on average across the evaluated datasets.
  • On the GSC and FSC datasets, the Conformer adapter achieves performance parity or surpasses full fine-tuning (FFT), while using only 0.29% (Pfeiffer configuration) or 0.59% (Houlsby configuration) of the parameters.
  • For audio classification tasks (ESC, US8K), the Conformer adapter performs on par with or slightly better than the bottleneck adapter and can even outperform FFT on US8K.
  • In few-shot learning scenarios, the Conformer adapter also shows strong performance, particularly on speech tasks.
  • The Conformer adapter demonstrates scalability, maintaining its advantage over other methods as the number of trainable parameters increases.
  • The Conformer adapter can be effectively applied to other pre-trained models, such as Wav2Vec 2.0, showing its generalizability.

Limitations and Future Work

  • Dataset sensitivity: The paper notes that the optimal kernel size for the Conformer adapter can vary depending on the dataset's complexity, with larger kernels being more beneficial for harder tasks.
  • Parameter allocation: While the Conformer adapter shows good scaling, the paper mentions that for FSC, LoRA required significantly more parameters (around 900K) to achieve similar performance to the Conformer adapter (around 100K). This suggests further exploration into optimal parameter allocation strategies might be beneficial.
  • Future work: The authors suggest investigating the role of kernel size further and exploring the application of the Conformer adapter to other pre-trained models and tasks. They also implicitly suggest that the architecture of adapters can be further optimized for specific domains.

Summary updated

Abstract

Parameter-efficient transfer learning (PETL) methods have emerged as a solid alternative to the standard full fine-tuning approach. They only train a few extra parameters for each downstream task, without sacrificing performance and dis- pensing with the issue of storing a copy of the pre-trained model for each task. For audio classification tasks, the Audio Spectrogram Transformer (AST) model shows im- pressive results. However, surprisingly, how to efficiently adapt it to several downstream tasks has not been tack- led before. In this paper, we bridge this gap and present a detailed investigation of common PETL methods for the adaptation of the AST model to audio/speech tasks. Fur- thermore, we propose a new adapter design that exploits the convolution module of the Conformer model, leading to superior performance over the standard PETL approaches and surpassing or achieving performance parity with full fine-tuning by updating only 0.29% of the parameters. Fi- nally, we provide ablation studies revealing that our pro- posed adapter: 1) proves to be effective in few-shot efficient transfer learning, 2) attains optimal results regardless of the amount of the allocated parameters, and 3) can be applied to other pre-trained models. Our code is available at https: //github.com/umbertocappellazzo/PETL_AST.
BibTeX
@inproceedings{cappellazzo2024parameter,
  author = {Umberto Cappellazzo and Daniele Falavigna and Alessio Brutti and Mirco Ravanelli},
  title = {{PARAMETER-EFFICIENT TRANSFER LEARNING OF AUDIO SPECTROGRAM TRANSFORMERS}},
  booktitle = {MLSP},
  year = {2024},
  doi = {10.1109/mlsp58920.2024.10734776},
}
Figure