HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models

Bingshen Mu, Kun Wei, Qijie Shao, Yong Xu, Lei Xie

TL;DR

HDMoLE is a parameter-efficient fine-tuning framework that allows LLM-based speech recognition models to master multiple accents without "forgetting" how to recognize standard speech. By combining a Mixture of LoRA Experts (MoLE) with a two-tier routing system and flexible activation thresholds, it matches the performance of full model fine-tuning while training only 9.6% of the parameters.

What Problem Does This Paper Address?

Integrating Large Language Models (LLMs) into Automatic Speech Recognition (ASR) has led to massive leaps in performance. However, these models often struggle with "out-of-distribution" speech, such as heavy regional accents or noisy environments.

Researchers typically solve this using Supervised Fine-Tuning (SFT), but this approach has two major drawbacks:

  1. Computational Cost: Retraining massive LLM-based systems is expensive.
  2. Catastrophic Forgetting: Adapting a model to a specific accent often degrades its performance on general, standard speech.

While Mixture of Experts (MoE) and LoRA (Low-Rank Adaptation) are common tools to mitigate these issues, standard MoE often suffers from "router collapse" (where only a few experts do all the work) and rigid selection strategies (like always picking exactly the top 2 experts) that don't adapt well to different layers of a neural network.

Methods

The authors introduce HDMoLE, which stands for Hierarchical routing and Dynamic thresholds for Mixture of LoRA Experts. The system consists of three main innovations:

  1. Mixture of LoRA Experts (MoLE): Instead of one giant weight update, the model uses several small, specialized LoRA adapters (experts). This keeps the original pre-trained weights frozen and efficient.
  2. Hierarchical Routing: To ensure experts actually specialize in specific accents, the authors use two routers:
  3. Global Router: A frozen, pre-trained Accent Recognition (AR) model. This provides an explicit "hint" about which accent is being spoken, forcing a clear correspondence between experts and domains.
  4. Local Router: A trainable layer that allows experts to collaborate implicitly based on the specific features of the hidden speech data.
  5. Dynamic Thresholds: Standard MoE uses a "Top-K" strategy (e.g., always pick the best 2 experts). HDMoLE replaces this with a learnable threshold. If an expert's weight is too low, it's discarded, and its weight is redistributed. This allows the model to dynamically decide how many experts are needed for a specific audio clip at each layer.

Datasets

The researchers evaluated HDMoLE on two primary datasets:

  • KeSpeech: A massive multi-accent Mandarin dataset featuring 1,542 hours of speech from 34 Chinese cities, covering eight major accent groups. This served as the "target domain."
  • AISHELL-2: A standard Mandarin dataset used to measure "catastrophic forgetting" in the "source domain."

The base model used was the Hubert+Baichuan2 architecture, a competitive LLM-based ASR framework.

Key Insights and Ideas

  • Solving Expert Imbalance: By using a pre-trained Accent Recognition model as a global router, the authors solved the common MoE problem where the router gets "lazy" and only picks a few favorite experts. The AR model forces the system to utilize accent-specific experts.
  • Layer-wise Intelligence: The "Dynamic Threshold" revealed a fascinating trend: lower layers of the model tend to activate more experts to handle complex, general speech features, while higher layers activate fewer, more specialized experts. A static Top-K approach would have missed this nuance.
  • The Best of Both Worlds: HDMoLE achieves a "Goldilocks" zone—it gains the specialized knowledge of full fine-tuning but retains the stability and efficiency of LoRA.

Key Results

  • Accuracy: On the multi-accent KeSpeech dataset, HDMoLE achieved a Character Error Rate (CER) of 16.58%, coming very close to the 15.64% achieved by full fine-tuning (the "topline").
  • Efficiency: HDMoLE only required 4.92 million trainable parameters—just 9.6% of the 51 million parameters required for full fine-tuning of the projector module.
  • Minimal Forgetting: While full fine-tuning caused the error rate on standard Mandarin (AISHELL-2) to jump from 3.50% to 4.91%, HDMoLE kept it at a much lower 3.69%, proving it preserves general knowledge far better.

Limitations and Future Work

The authors focused their implementation on the projector module (the bridge between the speech encoder and the LLM). While highly effective, they acknowledge that they haven't yet explored applying HDMoLE across the entire LLM-based ASR pipeline.

Future work will involve scaling this method to all linear layers within the LLM itself to see if even greater gains in accent robustness can be achieved. Additionally, the system currently relies on a pre-trained AR model; exploring ways to learn these domains unsupervised could be a promising next step.

Summary updated

Abstract

Recent advancements in integrating Large Language Models (LLM) with automatic speech recognition (ASR) have performed remarkably in general domains. While supervised fine-tuning (SFT) of all model parameters is often employed to adapt pre-trained LLM-based ASR models to specific domains, it imposes high computational costs and notably reduces their performance in general domains. In this paper, we propose a novel parameter-efficient multi-domain fine-tuning method for adapting pre-trained LLM-based ASR models to multi-accent domains without catastrophic forgetting named \textit{HDMoLE}, which leverages hierarchical routing and dynamic thresholds based on combining low-rank adaptation (LoRA) with the mixer of experts (MoE) and can be generalized to any linear layer. Hierarchical routing establishes a clear correspondence between LoRA experts and accent domains, improving cross-domain collaboration among the LoRA experts. Unlike the static Top-K strategy for activating LoRA experts, dynamic thresholds can adaptively activate varying numbers of LoRA experts at each MoE layer. Experiments on the multi-accent and standard Mandarin datasets demonstrate the efficacy of HDMoLE. Applying HDMoLE to an LLM-based ASR model projector module achieves similar performance to full fine-tuning in the target multi-accent domains while using only 9.6% of the trainable parameters required for full fine-tuning and minimal degradation in the source general domain.
BibTeX
@inproceedings{mu2025hdmole,
  author = {Bingshen Mu and Kun Wei and Qijie Shao and Yong Xu and Lei Xie},
  title = {{HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888133},
}
Figure