Introduction
Deploying state-of-the-art Automatic Speech Recognition (ASR) models like Whisper or Conformer on resource-constrained edge devices—such as smartphones and smart home assistants—presents a significant engineering challenge. While quantization (reducing numerical precision) can shrink these models, it typically results in a substantial loss of transcription accuracy.
This paper introduces SAML (Speaker Adaptive Mixture of LoRA Experts), a method that combines the parameter efficiency of Low-Rank Adaptation (LoRA) with the dynamic capacity of Mixture-of-Experts (MoE). SAML enables high-performance ASR on the edge by significantly compressing models while simultaneously improving accuracy for specific users through personalized adaptation.
The SAML Framework
SAML addresses the limitations of standard compression and adaptation techniques through a specialized architecture and a three-stage pipeline.
1. LoRA Experts and Dynamic Routing
Traditional MoE architectures are often too large for edge devices because their "experts" are dense, high-parameter layers. SAML replaces these with multiple lightweight LoRA modules. A small "router" network analyzes the input speech in real-time to determine how to weight the contributions of each LoRA expert. This dynamic approach allows the model to capture the variability of a speaker's voice more effectively than a single, static LoRA module.
2. The Personalised Quantised Model (PQM) Pipeline
The authors integrate SAML into a three-stage framework:
- Stage 1: Quantization: The base model is compressed using 4-bit NormalFloat (NF4) quantization.
- Stage 2: Pretraining: The router and LoRA experts are trained on a large, multi-speaker dataset to create a robust foundation.
- Stage 3: Adaptation: The system is fine-tuned on a small amount of data from a specific target speaker to specialize the model.
To maintain computational efficiency, the system sums the expert matrices before performing the final matrix multiplication, rather than executing multiple separate multiplications.
Experimental Results
The researchers evaluated SAML using the LibriSpeech (audiobook) and TED-LIUM 3 (TED talks) benchmarks. The results demonstrate that SAML provides a "best of both worlds" solution for model size and performance.
Massive Model Compression
SAML achieved approximately a 7x reduction in model size. For example, the Whisper-base model was compressed from 277.8MB to approximately 40MB, making it significantly more viable for local deployment on mobile hardware.
Superior Accuracy and Quantization Recovery
Strikingly, the adapted 4-bit models actually outperformed the original, full-precision (32-bit) models for the target speakers:
- Whisper: Achieved a 29.1% relative reduction in Word Error Rate (WER).
- Conformer: Achieved a 31.1% relative reduction in WER.
This suggests that speaker adaptation acts as a "cure" for quantization, effectively recovering and exceeding the information lost during the compression process.
Key Insights
- Dynamic vs. Static: In every test case, the SAML (Mixture of LoRA) approach outperformed standard single-LoRA adaptation. t-SNE visualizations of speaker features confirmed that SAML achieves a much clearer separation of voice characteristics, providing a richer representation than static modules.
- Layer Pruning: The authors observed that in certain layers, the router would "collapse" and consistently select only one expert. They discovered that these redundant experts and their routers could be pruned with zero loss in performance, further optimizing the model.
- Expert Scaling: While increasing the number of experts improves performance, the gains begin to level off after 10 experts, suggesting an efficiency "sweet spot" for edge applications.
Conclusion and Future Work
SAML demonstrates that high-performance ASR does not have to be sacrificed for model efficiency. By leveraging dynamic LoRA experts within a quantized framework, the authors achieved significant compression and improved accuracy. Future research may explore applying SAML to even larger model architectures or extending the framework to other speech tasks, such as text-to-speech or emotion recognition.
Summary updated