Speech-FT: A Fine-tuning Strategy for Enhancing Speech Representation Models Without Compromising Generalization Ability

Tzu-Quan Lin, Wei-Ping Huang, Hao Tang, Hung-yi LeearXiv 2025 · arXiv · OpenReview

Abstract

Speech representation models are highly effective at extracting general features for various tasks. While fine-tuning can enhance these representations for specific applications, it often compromises their generalization ability. To address this challenge, we propose Speech-FT, a fine-tuning strategy for speech representation models that leverages model merging to preserve generalization ability while still benefiting from fine-tuning. Speech-FT is effective across different fine-tuning scenarios and is compatible with various types of speech representation models, providing a versatile solution. Speech-FT offers an efficient and practical approach to further improving general speech representations after pre-training.

Introduction

Fine-tuning is the standard paradigm for adapting massive pre-trained models to specific downstream goals, such as speaker identification or emotion recognition. However, this process typically induces representational drift, where the model specializes in a target task at the expense of the general-purpose features learned during self-supervised pre-training. While existing solutions like LoRA or weight-space regularization attempt to mitigate this by keeping weights close to the original state, this paper argues that even small weight changes can lead to significant feature drift. Speech-FT addresses this by focusing on preserving robust representations through a novel two-stage framework.

Methodology: The Speech-FT Framework

Speech-FT introduces a simple yet effective two-stage approach to fine-tuning that balances task-specific adaptation with broad generalization.

  1. Stable Fine-tuning (Stable-FT): To prevent the transformer backbone from being disrupted by gradients from a randomly initialized task head, the process begins with a "Head-Start" phase. For the first 10% of training steps, only the task-specific prediction head is updated while the backbone remains frozen. Additionally, the convolutional front-end (downsampling module) is kept frozen throughout to preserve low-level audio pattern extraction.
  2. Weight-space Interpolation: After fine-tuning, the resulting weights ($\theta'$) are not used in isolation. Instead, they are merged with the original pre-trained weights ($\theta_0$) using linear interpolation: $\hat{\theta} = (1 - \alpha) \cdot \theta_0 + \alpha \cdot \theta'$. Empirical results suggest a scaling factor of $\alpha = 0.25$ provides an optimal balance.
Figure 1: The three-step workflow of Speech-FT, including Stable Fine-tuning, Weight-space Interpolation, and evaluation on the SUPERB benchmark.

Experimental Setup

The researchers evaluated Speech-FT across a diverse set of models, including HuBERT, wav2vec 2.0, DeCoAR 2.0, and WavLM Base+. The evaluation utilized the SUPERB (Speech Processing Universal PERformance Benchmark) to measure generalization across ten simultaneous speech tasks, alongside specific supervised tasks like ASR, Phoneme Recognition (PR), and Speaker Identification (SID).

Key Results and Discussion

Speech-FT consistently outperforms standard fine-tuning and Stage-1 (Stable-FT) alone in maintaining general-purpose utility. As shown in the results table, when a model is fine-tuned on a specific task (e.g., ASR on TED-LIUM), Speech-FT significantly improves the overall SUPERB score compared to the pre-trained baseline and the intermediate Stable-FT stage.

Figure 4: Performance comparison across various fine-tuning tasks. Speech-FT (the full two-stage method) consistently achieves higher SUPERB scores than Stable-FT alone, indicating superior generalization.

Insights on Feature Connectivity

The success of Speech-FT is theoretically grounded in Layerwise Linear Feature Connectivity (LLFC). The authors demonstrate that internal representations evolve almost linearly during fine-tuning. Consequently, weight-space interpolation acts as a restorative force, filtering out the "noise" of over-specialization while retaining the task-specific "signal." This allows Speech-FT to maintain higher feature similarity to the original model than traditional regularization methods, even when weight changes are larger.

Conclusion

Speech-FT provides a practical and efficient strategy for enhancing speech representation models. By combining stabilized training with weight-space interpolation, it allows models to gain task-specific knowledge while simultaneously improving—rather than compromising—their general-purpose performance. Future work may extend this merging-based approach to more complex domains like Spoken Language Models (SLMs), where maintaining generative capabilities during task adaptation is a critical challenge.

Summary updated

Abstract

Speech representation models are highly effective at extracting general features for various tasks. While fine-tuning can enhance these representations for specific applications, it often compromises their generalization ability. To address this challenge, we propose Speech-FT, a fine-tuning strategy for speech representation models that leverages model merging to preserve generalization ability while still benefiting from fine-tuning. Speech-FT is effective across different fine-tuning scenarios and is compatible with various types of speech representation models, providing a versatile solution. Speech-FT offers an efficient and practical approach to further improving general speech representations after pre-training.
BibTeX
@misc{lin2025speech,
  author = {Tzu-Quan Lin and Wei-Ping Huang and Hao Tang and Hung-yi Lee},
  title = {{Speech-FT: A Fine-tuning Strategy for Enhancing Speech Representation Models Without Compromising Generalization Ability}},
  year = {2025},
  eprint = {2502.12672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.12672},
}
Figure