Abstract
Automatic Speech Recognition (ASR) systems must handle significant acoustic variability across different environments and speaker accents. While Mixture-of-Experts (MoE) models provide a scalable architecture by activating only a subset of parameters, traditional routers often lack the global context necessary for optimal expert selection. This paper introduces SpeechMoE2, which enhances the routing mechanism by incorporating global domain and accent embeddings. Evaluated on a 150,000-hour Mandarin dataset, SpeechMoE2 achieves up to 17.7% relative improvement in Character Error Rate (CER) over baseline models while maintaining constant inference computation costs.
Introduction
Large-scale speech recognition systems face the challenge of immense diversity in input data, ranging from background noise and reverberation to regional dialects. MoE models have emerged as a popular solution, allowing for massive model capacity without a proportional increase in computational cost (FLOPs). In these models, a router decides which specialized "expert" sub-networks should process specific speech frames. However, previous routers primarily utilized local, frame-level phonetic information, making them "blind" to the broader global context of the speaker and environment. SpeechMoE2 addresses this limitation by providing the router with "global eyes" through integrated domain and accent information.
Methods
SpeechMoE2 introduces a redesigned router architecture and a multi-task learning objective to improve expert dispatching.
Global Feature Integration
Unlike previous models that relied on local grapheme embeddings, the SpeechMoE2 router receives domain embeddings (representing the acoustic environment) and accent embeddings (representing the speaker's dialect). These are concatenated with the output of the previous layer to calculate the activation probabilities for the experts.
Multi-Task Embedding Network
To generate these global features efficiently, the authors utilize a shared embedding network trained on three simultaneous tasks:
- CTC Loss: Ensures phonetic and grapheme accuracy.
- Domain Classification: Identifies the acoustic environment (e.g., clean, noisy, echo-cancelled).
- Accent Classification: Identifies regional accents (e.g., Sichuan, Hunan).
The researchers scaled these configurations using the FastMoE framework, testing models with up to 16 experts per layer.
Key Insights
- Context-Aware Routing: Routing is fundamentally a classification problem. By providing global context, the router can immediately identify the optimal experts for specific acoustic conditions, such as a particular accent in a noisy environment.
- Specialization through Multi-Tasking: The multi-task learning objective forces the embedding network to produce highly discriminative vectors. This "extra knowledge" guides experts to specialize in specific acoustic niches, improving overall model robustness.
- The "Free" Scaling Lunch: MoE architectures allow for increasing total parameter counts (e.g., from 107M to 500M) to improve accuracy while keeping the inference FLOPs identical, as only the top-performing experts are activated for any given frame.
Results
SpeechMoE2 demonstrates significant performance gains across various test conditions. In multi-domain evaluations, the model achieved relative CER improvements of 1.6% to 4.8% over the baseline SpeechMoE. The improvements were even more pronounced in multi-accent testing, showing gains between 1.9% and 17.7%.
As shown in the results, increasing the number of experts from 2 to 16 provides a consistent performance boost. For instance, on the "Hunan" accent test set, the CER dropped significantly as the model scaled, all while maintaining a constant 2.3B FLOPs.
Conclusion and Future Work
SpeechMoE2 proves that incorporating global context into the MoE routing mechanism significantly enhances ASR performance, particularly for diverse accents and environments. While the current study utilizes a sequential memory and self-attention backbone, future work will explore more modern architectures like the Conformer. Additionally, the researchers plan to investigate more sophisticated routing strategies beyond the current "top-1" approach to further optimize expert utilization.
Summary updated