Resource-Efficient Transfer Learning from Speech Foundation Model Using Hierarchical Feature Fusion

Zhouyuan Huo, Khe Chai Sim, Bo Li, Dongseong Hwang, Tara N. Sainath, Trevor Strohman

TL;DR

Researchers at Google have developed a "Hierarchical Feature Fusion" (HFF) method that allows massive speech foundation models to be adapted to specific tasks with extreme efficiency. By treating the foundation model as a frozen feature extractor and merging information from multiple internal layers, they achieved the same accuracy as full model fine-tuning while using 97% fewer trainable parameters and cutting training time by 53%.

What Problem Does This Paper Address?

Foundation models (like Wav2vec 2.0 or HuBERT) are pre-trained on massive amounts of unlabeled data and then "fine-tuned" for specific tasks like speech recognition. However, these models are often enormous (billions of parameters).

Standard fine-tuning is problematic for two reasons:

  1. Resource Intensity: Updating every parameter for every new task is computationally expensive and requires massive storage.
  2. The Backpropagation Bottleneck: Existing "parameter-efficient" methods (like Adapters or BitFit) only update a small fraction of the model. However, because these trainable bits are scattered throughout the network, you still have to run "backpropagation" through the entire giant model to calculate gradients. This means memory usage and training speeds remain sluggish despite having fewer "trainable" parameters.

Furthermore, most methods only use the output of the very last layer, ignoring the "goldmine" of acoustic and contextual information stored in the middle layers of the network.

Methods

The authors propose moving away from updating the foundation model itself. Instead, they treat it as a frozen feature extractor and focus on how to best combine the "knowledge" sitting in different layers.

  • Linear Feature Fusion: A baseline approach where outputs from multiple layers are concatenated and projected into a new space.
  • Hierarchical Feature Fusion (HFF): The core innovation. Instead of just stacking layers, the model fuses them in pairs (hierarchically).
  • HFF-balanced (HFF-b): Projects and merges neighboring pairs of layers equally.
  • HFF-unbalanced (HFF-ub): Merges layers from the bottom-up or top-down, prioritizing specific levels of abstraction.
  • Feature Projectors (FPs): Small, trainable fully-connected networks are placed at each layer's output before fusion. This allows the system to "pick and choose" which features from which layers are most relevant for the task at hand.
  • Hybrid Approach: They eventually combine HFF with Adapters (small trainable modules) placed only in the top layers to further refine performance without re-introducing the full backpropagation bottleneck.

Datasets

  • Pre-training: 800,000 hours of unsupervised, de-identified English YouTube audio.
  • Downstream Task (Fine-tuning/Evaluation): 5,000 hours of labeled "Voice Search" (VS) data. This is a standard benchmark for real-world automatic speech recognition (ASR) performance.

Key Insights and Ideas

  • The "Middle Layer" Sweet Spot: The authors confirmed that for speech recognition, the middle layers of a foundation model are actually the most useful. The bottom layers contain raw acoustic data, and the top layers often become too specialized during pre-training. HFF allows the model to exploit these rich middle-layer representations.
  • Freezing is Faster: By freezing the foundation model and only training the fusion layers on top, the researchers eliminated the need to backpropagate through the 600M+ parameter "trunk." This is the primary driver behind the 53% increase in training speed.
  • Hierarchy Matters: Hierarchical fusion (HFF-b) consistently outperformed simple linear concatenation, suggesting that the relationship between different levels of neural abstraction is non-linear and benefits from structured merging.

Key Results

The researchers compared their HFF method against "Fine-tune all" (the gold standard for accuracy) and other popular efficiency methods:

  • Accuracy Parity: When combining HFF-b with Adapters at the top layers, they achieved a 5.5% Word Error Rate (WER)—exactly matching the performance of fine-tuning the entire 600M parameter model.
  • Efficiency Gains: This parity was achieved using only 18.6M trainable encoder parameters (compared to 606.6M in the full model) and significantly lower computational memory.
  • Superiority over Baselines: HFF-b (7.0% WER) significantly outperformed BitFit (6.5% but much slower) and "Fine-tuning the highest layer" (15.8% WER), proving that multi-layer fusion is critical.

Limitations and Future Work

  • Task Specificity: The study focused primarily on Automatic Speech Recognition (ASR). It is unclear if HFF would be as effective for non-transcription tasks like emotion recognition or speaker identification, where different layers might be more relevant.
  • Architecture Dependency: The experiments were conducted on Conformer-based foundation models. While Conformers are state-of-the-art in speech, the "hierarchical" benefits might vary for other architectures like standard Transformers or CNNs.
  • Future Direction: The authors suggest investigating the fusion of features from multiple different pre-trained models (ensembling) using this hierarchical approach to see if complementary pre-training strategies can be merged efficiently.

Summary updated

Abstract

Self-supervised pre-training of a speech foundation model, followed by supervised fine-tuning, has shown impressive quality improvements on automatic speech recognition (ASR) tasks. Fine-tuning separate foundation models for many downstream tasks are expensive since the foundation model is usually very big. Parameter-efficient fine-tuning methods (e.g. adapter, sparse update methods) offer an alternative paradigm where a small set of parameters are updated to adapt the foundation model to new tasks. However, these methods still suffer from a high computational memory cost and slow training speed because they require backpropagation through the entire neural network at each step. In the paper, we analyze the performance of features at different layers of a foundation model on the speech recognition task and propose a novel hierarchical feature fusion method for resource-efficient transfer learning from speech foundation models. Experimental results show that the proposed method can achieve better performance on speech recognition task than existing algorithms with fewer number of trainable parameters, less computational memory cost and faster training speed. After combining with Adapters at all layers, the proposed method can achieve the same performance as fine-tuning the whole model with 97% fewer trainable encoder parameters and 53% faster training speed.
BibTeX
@inproceedings{huo2023resource,
  author = {Zhouyuan Huo and Khe Chai Sim and Bo Li and Dongseong Hwang and Tara N. Sainath and Trevor Strohman},
  title = {{Resource-Efficient Transfer Learning from Speech Foundation Model Using Hierarchical Feature Fusion}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096799},
}

Loading related content...

Figure