Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization

TL;DR

Fine-tuning self-supervised speech representation models on a specific task usually ruins their ability to generalize to other tasks. To solve this "representational drift," Speech-FT introduces a simple two-stage framework: a stabilized fine-tuning phase followed by weight-space interpolation (model merging) with the original pre-trained model. This approach simultaneously boosts performance on the target task and enhances cross-task generalization across a wide variety of speech applications.

What Problem Does This Paper Address?

Self-supervised speech models (like HuBERT or WavLM) are fantastic foundation models because they learn general speech representations from massive amounts of unlabeled audio. However, when practitioners fine-tune these models on a specific downstream task (like Automatic Speech Recognition), the model over-specializes. This causes "representational drift," where the model forgets the general features it learned during pre-training, severely degrading its performance on unrelated tasks (e.g., speaker identification or emotion recognition).

Traditional fixes like weight-space regularization (penalizing large weight changes) or parameter-efficient methods like LoRA try to keep the model close to its original state in weight space. However, the authors show that these constraints fail to maintain actual feature similarity, still resulting in a loss of generalization.

Methods

The authors propose Speech-FT (Speech Fine-Tuning), a clean, two-stage framework:

1. Stable Fine-Tuning (Stable-FT): To minimize early representational drift, the framework first freezes the low-level convolutional downsampling module (which captures fundamental audio frequencies). It then trains only the newly attached task prediction head for the first 10% of steps (a warm-up phase) before unfreezing the rest of the representation model.
2. Weight-Space Interpolation: After fine-tuning, instead of throwing away the pre-trained model, Speech-FT performs a linear interpolation between the pre-trained weights ($\theta_0$) and the fine-tuned weights ($\theta'$): $$\hat{\theta} = (1 - \alpha) \cdot \theta_0 + \alpha \cdot \theta'$$ By setting $\alpha = 0.25$, the model effectively injects task-specific knowledge while pulling the overall feature space back toward the robust, generalized pre-training geometry.

The complete pipeline is illustrated below:

Expand

Datasets

The framework was evaluated across a diverse set of speech processing datasets:

• Fine-tuning Tasks: TED-LIUM (ASR), TIMIT (Phoneme Classification), LibriSpeech (ASR, Phoneme Recognition, Speaker Identification), CREMA-D (Emotion Recognition), VoxCeleb1 (Speaker Identification), and IEMOCAP (Emotion Recognition).
• Cross-Lingual/Unsupervised Adaptation: AISHELL-3 (Mandarin speech corpus) and the Spanish subset of Multilingual LibriSpeech.
• Evaluation Benchmark: The SUPERB (and MiniSUPERB) benchmark, which measures cross-task generalization across four core dimensions: Phoneme Recognition (PR), Speaker Identification (SID), Emotion Recognition (ER), and Slot Filling (SF).

Key Insights and Ideas

• Weight Distance $\neq$ Feature Similarity: This is the core "aha" moment of the paper. Methods like LoRA or weight regularization successfully keep the model close to the pre-trained model in weight space, but they still cause massive distortion in feature space. Speech-FT actually allows larger weight updates during fine-tuning but restores exceptionally high feature similarity after the interpolation step.
• Low-Impact Subspaces: The success of Speech-FT empirically proves the existence of "invariant subspaces" in speech models—regions in weight space where substantial parameter updates result in minimal functional changes to the underlying representations.
• Layerwise Linear Feature Connectivity (LLFC): The authors demonstrate that internal representations evolve almost linearly along the interpolation path. This means merging weights linearly translates to a clean, predictable blend of features without destroying representation geometry.

Key Results

• Consistent SUPERB Wins: Speech-FT consistently outperformed both the original pre-trained models and standard fine-tuning baselines across all tasks.
• ASR Fine-Tuning Boost: When fine-tuning HuBERT on TED-LIUM ASR, Speech-FT reduced the phone error rate (PER) from 5.17% to 3.94%, lowered the word error rate (WER) from 6.38% to 5.75%, and boosted speaker identification accuracy from 81.86% to 84.11%.
• Cross-Lingual Generalization: In unsupervised cross-lingual adaptation (continuing pre-training on Mandarin AISHELL-3), standard fine-tuning caused catastrophic forgetting of English tasks (the SUPERB score dropped by over 80 points). Speech-FT successfully adapted to Mandarin while fully preserving English performance.
• Model Agnostic: The approach proved highly effective across multiple architectures, including HuBERT, wav2vec 2.0, DeCoAR 2.0, and the state-of-the-art WavLM Base+.

Limitations and Future Work

• Hyperparameter Tuning: Although $\alpha = 0.25$ generalized remarkably well across most experiments, finding the absolute optimal interpolation factor might still require some task-specific validation.
• Future Directions: The authors suggest exploring the applicability of Speech-FT to broader domains, such as generative spoken language modeling (SpeechLM) and more complex multi-task model merging strategies (like TIES-Merging) across massive task pools.