Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning

Hongli Yang, Yizhou Peng, Hao Huang, Sheng LiInterspeech 2025

TL;DR

This paper explores parameter-efficient methods, specifically Soft Prompt Tuning (SPT), to adapt the powerful Whisper ASR model for code-switching (CS) scenarios without sacrificing performance on existing languages. They introduce SPT4ASR, a novel combination of SPT variants, demonstrating competitive results with significantly fewer trainable parameters than full fine-tuning.

What Problem Does This Paper Address?

Large-scale multilingual Automatic Speech Recognition (ASR) models like OpenAI's Whisper have shown remarkable capabilities across many languages. However, adapting these massive models to specialized, low-resource scenarios, such as code-switching (CS) – where speakers switch between languages within a single utterance or conversation – presents significant challenges. Traditional full fine-tuning (FFT) is computationally expensive and risks "catastrophic forgetting," where the model loses performance on its original training data. Parameter-efficient fine-tuning (PEFT) methods offer a solution by adapting models with far fewer trainable parameters, but their effectiveness in the complex domain of CS ASR, especially in comparison to each other and in preserving original capabilities, is not well-understood. This paper aims to fill that gap by systematically evaluating various SPT techniques for CS ASR.

Methods

The authors investigate several parameter-efficient methods for adapting Whisper:

  • Full Fine-Tuning (FFT): This is the standard approach where all parameters of the Whisper model are updated during training on the new dataset. While effective, it's computationally intensive and prone to catastrophic forgetting.
  • Soft Prompt Tuning (SPT): This method keeps the pre-trained Whisper model parameters frozen and instead trains a small set of "soft prompt" embeddings. These prompts are prepended to the input sequences (encoder or decoder) to guide the model's behavior.
  • Vanilla SPT: Soft prompts are added to the initial input features.
  • Deep Prompt Tuning (DPT): Soft prompts are inserted at multiple intermediate layers of the Whisper model, allowing for more nuanced control.
  • Residual Prompt Tuning (ResPT): This approach uses a reparameterization technique with residual connections to generate more robust and efficient prompts.
  • Language Prompt Tuning (LPT): This method leverages Whisper's language identification capabilities by encoding language IDs into prompt embeddings to improve language discrimination in CS scenarios.
  • SPT4ASR: This is the paper's novel contribution, a combination of different SPT variants designed to maximize performance in CS ASR.
  • LoRA (Low-Rank Adaptation): Included as a benchmark PEFT method, LoRA injects trainable low-rank matrices into specific layers of the model.

Datasets

The primary datasets used for training and evaluating code-switching ASR performance were:

  • SEAME: A conversational Mandarin-English dataset from Southeast Asia (Malaysia and Singapore).
  • ASRU2019: A Mandarin-English CS dataset from Mainland China.

To evaluate the mitigation of catastrophic forgetting, the authors also used:

  • Aishell1: A Chinese ASR dataset.
  • LibriSpeech: A dataset for English ASR.
  • SPREDS-U1: A Korean language evaluation dataset.

Key Insights and Ideas

  • Deep Prompt Tuning is Powerful: Inserting soft prompts at multiple layers (DPT) significantly improves performance over vanilla SPT, suggesting that finer-grained control within the model architecture is beneficial.
  • SPT4ASR Shows Promise: Combining various SPT techniques (SPT4ASR) yields the best results among SPT-based methods, demonstrating the synergy of different prompt tuning strategies.
  • Parameter Efficiency vs. Performance Trade-off: SPT4ASR achieves performance close to FFT but with drastically fewer trainable parameters, making it a more efficient alternative for adaptation.
  • LoRA Still Leads in Efficiency: While SPT4ASR is competitive, LoRA generally offers a better balance of performance and parameter efficiency, though SPT methods are a viable alternative.
  • Whole Model Tuning with Prompts is Effective: Surprisingly, integrating soft prompts with whole-model fine-tuning (not just the prompts) yielded superior results to standard FFT in some cases, suggesting a nuanced interaction.
  • SPT Mitigates Catastrophic Forgetting: Similar to LoRA, SPT methods effectively preserve performance on monolingual tasks, unlike FFT, highlighting their advantage in retaining pre-trained knowledge.

Key Results

  • SEAME & ASRU2019 Performance: SPT4ASR achieved competitive results on both SEAME and ASRU2019 datasets, significantly outperforming zero-shot Whisper performance and performing comparably to the ESPnet Conformer. For instance, for Whisper-medium, SPT4ASR (M3) achieved a devman MER of 13.12% and a devsge MER of 18.10% on SEAME, while FFT (M1) achieved 11.44% and 17.55% respectively.
  • Parameter Efficiency: SPT4ASR methods used around 3.74M parameters for Whisper-small, a fraction of the parameters used by FFT (240.58M) or whole-model SPT (240.69M).
  • Catastrophic Forgetting Mitigation: Table 4 shows that SPT4ASR (C5) and LoRA (C4) maintained performance close to the baseline Whisper Medium model (C2) on monolingual tasks, whereas FFT (C3) showed significant degradation.

Limitations and Future Work

  • LoRA's Superiority: The paper acknowledges that LoRA generally outperforms SPT4ASR in terms of both performance and parameter efficiency.
  • Prompt Length Sensitivity: The performance of Vanilla SPT was found to be sensitive to prompt length, with optimal lengths varying.
  • Future Work: The authors suggest that soft prompt-based methods offer a compelling trade-off between adaptability and resource usage and warrant continued investigation in multilingual ASR. Further exploration into combining SPT with other PEFT techniques or optimizing prompt placement could yield even better results.

Summary updated

BibTeX
@inproceedings{yang2025adapting,
  author = {Hongli Yang and Yizhou Peng and Hao Huang and Sheng Li},
  title = {{Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure