Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion

Zhouyuan Huo, Khe Chai Sim, Bo Li, Dongseong Hwang, Tara N. Sainath, Trevor Strohman

Abstract

Self-supervised pre-training of a speech foundation model, followed by supervised fine-tuning, has shown impressive quality improvements on automatic speech recognition (ASR) tasks. Fine-tuning separate foundation models for many downstream tasks are expensive since the foundation model is usually very big. Parameter-efficient fine-tuning methods (e.g. adapter, sparse update methods) offer an alternative paradigm where a small set of parameters are updated to adapt the foundation model to new tasks. However, these methods still suffer from a high computational memory cost and slow training speed because they require backpropagation through the entire neural network at each step. In the paper, we analyze the performance of features at different layers of a foundation model on the speech recognition task and propose a novel hierarchical feature fusion method for resource-efficient transfer learning from speech foundation models. Experimental results show that the proposed method can achieve better performance on speech recognition task than existing algorithms with fewer number of trainable parameters, less computational memory cost and faster training speed. After combining with Adapters at all layers, the proposed method can achieve the same performance as fine-tuning the whole model with 97% fewer trainable encoder parameters and 53% faster training speed.

Introduction

Foundation models like Wav2vec 2.0 or HuBERT are pre-trained on vast unlabeled datasets and subsequently fine-tuned for downstream tasks such as automatic speech recognition (ASR). While effective, these models often contain billions of parameters, making standard fine-tuning computationally prohibitive. Updating every parameter for each new task requires massive storage and compute.

Furthermore, existing parameter-efficient methods like Adapters or BitFit, which only update a fraction of the model, still face a "backpropagation bottleneck." Because trainable modules are distributed throughout the network, gradients must be calculated through the entire frozen trunk, maintaining high memory usage and slow training speeds. Additionally, most approaches rely solely on the final layer's output, neglecting the rich acoustic and contextual information embedded in the model's intermediate layers.

Hierarchical Feature Fusion (HFF)

To address these challenges, we propose treating the foundation model as a frozen feature extractor. Instead of updating the model's internal weights, we focus on optimally combining representations from multiple layers.

The core innovation is Hierarchical Feature Fusion (HFF). Unlike simple linear concatenation, HFF merges layer outputs in pairs through structured hierarchies. We explore two variants:

  • HFF-balanced (HFF-b): Merges neighboring pairs of layers with equal weighting.
  • HFF-unbalanced (HFF-ub): Merges layers bottom-up or top-down to prioritize specific levels of abstraction.

Small, trainable Feature Projectors (FPs) are placed at each layer's output before fusion, allowing the system to learn which features are most relevant for the target task.

The proposed Hierarchical Feature Fusion (HFF) architectures: Balanced and Unbalanced variants.

Experimental Results and Insights

Experiments were conducted using a foundation model pre-trained on 800,000 hours of English YouTube audio, evaluated on a 5,000-hour Voice Search (VS) benchmark.

The "Middle Layer" Sweet Spot

A key finding is that middle layers are the most informative for ASR. Bottom layers capture raw acoustics, while top layers may become overly specialized to the pre-training objective. HFF effectively exploits these rich intermediate representations.

Word Error Rate (WER) across different layers, highlighting the superior performance of middle layers.

Efficiency and Performance

By freezing the foundation model trunk and only training the fusion layers, we eliminate backpropagation through the 600M+ parameter core. This resulted in a 53% increase in training speed.

When combining HFF-b with Adapters in the top layers, the system achieved a 5.5% Word Error Rate (WER), matching full-model fine-tuning while using 97% fewer trainable encoder parameters (18.6M vs 606.6M). HFF-b (7.0% WER) also significantly outperformed other efficient baselines like BitFit (6.5% but much slower) and single-layer fine-tuning (15.8% WER) in terms of the overall efficiency-accuracy trade-off.

Conclusion

Hierarchical Feature Fusion provides a highly efficient pathway for adapting massive speech foundation models. By leveraging intermediate layer knowledge and avoiding full-trunk backpropagation, HFF achieves state-of-the-art accuracy with minimal resource overhead. Future work will explore HFF's effectiveness across diverse tasks like emotion recognition and investigate the fusion of multiple complementary pre-trained models.

Summary updated

Abstract

Self-supervised pre-training of a speech foundation model, followed by supervised fine-tuning, has shown impressive quality improvements on automatic speech recognition (ASR) tasks. Fine-tuning separate foundation models for many downstream tasks are expensive since the foundation model is usually very big. Parameter-efficient fine-tuning methods (e.g. adapter, sparse update methods) offer an alternative paradigm where a small set of parameters are updated to adapt the foundation model to new tasks. However, these methods still suffer from a high computational memory cost and slow training speed because they require backpropagation through the entire neural network at each step. In the paper, we analyze the performance of features at different layers of a foundation model on the speech recognition task and propose a novel hierarchical feature fusion method for resource-efficient transfer learning from speech foundation models. Experimental results show that the proposed method can achieve better performance on speech recognition task than existing algorithms with fewer number of trainable parameters, less computational memory cost and faster training speed. After combining with Adapters at all layers, the proposed method can achieve the same performance as fine-tuning the whole model with $97\%$ fewer trainable encoder parameters and $53\%$ faster training speed.
BibTeX
@inproceedings{huo2023resource,
  author = {Zhouyuan Huo and Khe Chai Sim and Bo Li and Dongseong Hwang and Tara N. Sainath and Trevor Strohman},
  title = {{Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096799},
}
Figure