SpeechMoE2: Mixture-of-Experts Model with Improved Routing

Zhao You, Shulin Feng, Dan Su, Dong Yu

Abstract

Automatic Speech Recognition (ASR) systems must handle significant acoustic variability across different environments and speaker accents. While Mixture-of-Experts (MoE) models provide a scalable architecture by activating only a subset of parameters, traditional routers often lack the global context necessary for optimal expert selection. This paper introduces SpeechMoE2, which enhances the routing mechanism by incorporating global domain and accent embeddings. Evaluated on a 150,000-hour Mandarin dataset, SpeechMoE2 achieves up to 17.7% relative improvement in Character Error Rate (CER) over baseline models while maintaining constant inference computation costs.

Introduction

Large-scale speech recognition systems face the challenge of immense diversity in input data, ranging from background noise and reverberation to regional dialects. MoE models have emerged as a popular solution, allowing for massive model capacity without a proportional increase in computational cost (FLOPs). In these models, a router decides which specialized "expert" sub-networks should process specific speech frames. However, previous routers primarily utilized local, frame-level phonetic information, making them "blind" to the broader global context of the speaker and environment. SpeechMoE2 addresses this limitation by providing the router with "global eyes" through integrated domain and accent information.

Methods

SpeechMoE2 introduces a redesigned router architecture and a multi-task learning objective to improve expert dispatching.

Global Feature Integration

Unlike previous models that relied on local grapheme embeddings, the SpeechMoE2 router receives domain embeddings (representing the acoustic environment) and accent embeddings (representing the speaker's dialect). These are concatenated with the output of the previous layer to calculate the activation probabilities for the experts.

Multi-Task Embedding Network

To generate these global features efficiently, the authors utilize a shared embedding network trained on three simultaneous tasks:

  1. CTC Loss: Ensures phonetic and grapheme accuracy.
  2. Domain Classification: Identifies the acoustic environment (e.g., clean, noisy, echo-cancelled).
  3. Accent Classification: Identifies regional accents (e.g., Sichuan, Hunan).
Architecture comparison between (a) the original SpeechMoE and (b) the proposed SpeechMoE2, highlighting the integration of domain and accent classification for improved routing.

The researchers scaled these configurations using the FastMoE framework, testing models with up to 16 experts per layer.

Key Insights

  • Context-Aware Routing: Routing is fundamentally a classification problem. By providing global context, the router can immediately identify the optimal experts for specific acoustic conditions, such as a particular accent in a noisy environment.
  • Specialization through Multi-Tasking: The multi-task learning objective forces the embedding network to produce highly discriminative vectors. This "extra knowledge" guides experts to specialize in specific acoustic niches, improving overall model robustness.
  • The "Free" Scaling Lunch: MoE architectures allow for increasing total parameter counts (e.g., from 107M to 500M) to improve accuracy while keeping the inference FLOPs identical, as only the top-performing experts are activated for any given frame.

Results

SpeechMoE2 demonstrates significant performance gains across various test conditions. In multi-domain evaluations, the model achieved relative CER improvements of 1.6% to 4.8% over the baseline SpeechMoE. The improvements were even more pronounced in multi-accent testing, showing gains between 1.9% and 17.7%.

Comparison of model parameters, FLOPs, and performance across different test sets. Increasing the number of experts consistently reduces CER while maintaining constant computational cost.

As shown in the results, increasing the number of experts from 2 to 16 provides a consistent performance boost. For instance, on the "Hunan" accent test set, the CER dropped significantly as the model scaled, all while maintaining a constant 2.3B FLOPs.

Conclusion and Future Work

SpeechMoE2 proves that incorporating global context into the MoE routing mechanism significantly enhances ASR performance, particularly for diverse accents and environments. While the current study utilizes a sequential memory and self-attention backbone, future work will explore more modern architectures like the Conformer. Additionally, the researchers plan to investigate more sophisticated routing strategies beyond the current "top-1" approach to further optimize expert utilization.

Summary updated

BibTeX
@inproceedings{you2022speechmoe,
  author = {Zhao You and Shulin Feng and Dan Su and Dong Yu},
  title = {{SpeechMoE2: Mixture-of-Experts Model with Improved Routing}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747065},
}
Figure