Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition

Wenxuan Wang, Guodong Ma, Yuke Li, Binbin Du

Abstract

Multilingual speech recognition for both monolingual and code-switching speech is a challenging task. Recently, based on the Mixture of Experts (MoE), many works have made good progress in multilingual and code-switching ASR, but present huge computational complexity with the increase of supported languages. In this work, we propose a computation-efficient network named Language-Routing Mixture of Experts (LR-MoE) for multilingual and code-switching ASR. LR-MoE extracts language-specific representations through the Mixture of Language Experts (MLE), which is guided to learn by a frame-wise language routing mechanism. The weight-shared frame-level language identification (LID) network is jointly trained as the shared pre-router of each MoE layer. Experiments show that the proposed method significantly improves multilingual and code-switching speech recognition performances over baseline with comparable computational efficiency.

Introduction

Multilingual Automatic Speech Recognition (ASR) faces significant hurdles, particularly in code-switching scenarios where speakers alternate between languages within a single utterance. While Mixture of Experts (MoE) architectures have shown promise by allowing specialized sub-networks to handle different languages, they often suffer from high computational costs and context loss during language transitions. Traditional MoE models typically compute representations for all experts or use complex routing that doesn't scale well. This paper introduces the Language-Routing Mixture of Experts (LR-MoE), designed to maintain high accuracy with constant computational complexity regardless of the number of languages supported.

Proposed Method: LR-MoE

The LR-MoE architecture optimizes the Transformer encoder by partitioning it into two distinct functional components: Shared Blocks and Mixture of Language Experts (MLE) Blocks.

The LR-MoE architecture featuring Shared Blocks and MLE Blocks with frame-wise routing.

Frame-wise Language Routing (FLR)

The core innovation is the Frame-wise Language Routing (FLR) mechanism. Unlike utterance-level routing, FLR utilizes a shared Language Identification (LID) gated network to predict the language for every individual audio frame (typically 10-25ms chunks).

  • Dynamic Routing: A Top-1 routing strategy ensures that each frame is processed by exactly one language-specific expert, keeping GFLOPs constant regardless of the total number of experts.
  • Joint Training: The LID router is trained multi-task style with Connectionist Temporal Classification (CTC) loss, ensuring precise alignment between the routing decisions and the acoustic content.
  • Strategic Placement: Research indicated that placing MLE layers in the middle-to-late stages of the encoder allows the shared blocks to first extract robust, language-agnostic features, which the experts then refine.

Experimental Results

The model was evaluated on Mandarin-English code-switching and a broader five-language multilingual set including Japanese and Korean.

Performance comparison of LR-MoE against various baselines in terms of parameters and accuracy (CER/WER).

Key Findings

  • Efficiency: The FLR-MoE model maintains the same computational footprint (55.4 GFLOPs) as a vanilla baseline, whereas traditional multi-encoder approaches require nearly triple the compute (146.9 GFLOPs).
  • Accuracy: On the Mandarin-English task, the LR-MoE (AED version) achieved a 4.7% Character Error Rate (CER) for Mandarin and a 9.6% Word Error Rate (WER) for English, surpassing the state-of-the-art Language-Aware Encoder (LAE).
  • LID Precision: The frame-wise router achieved an average LID accuracy of 99.4%, demonstrating its effectiveness in identifying language boundaries even during rapid code-switching.

Conclusion and Future Work

LR-MoE provides a scalable and efficient solution for complex multilingual ASR tasks. By leveraging frame-level routing, it achieves state-of-the-art performance without the typical "computational tax" of MoE models. Future research will focus on improving expert utilization beyond Top-1 routing and testing the architecture's limits with dozens of languages to further explore its scalability in global-scale ASR systems.

Summary updated

Abstract

Multilingual speech recognition for both monolingual and code-switching speech is a challenging task. Recently, based on the Mixture of Experts (MoE), many works have made good progress in multilingual and code-switching ASR, but present huge computational complexity with the increase of supported languages. In this work, we propose a computation-efficient network named Language-Routing Mixture of Experts (LR-MoE) for multilingual and code-switching ASR. LR-MoE extracts language-specific representations through the Mixture of Language Experts (MLE), which is guided to learn by a frame-wise language routing mechanism. The weight-shared frame-level language identification (LID) network is jointly trained as the shared pre-router of each MoE layer. Experiments show that the proposed method significantly improves multilingual and code-switching speech recognition performances over baseline with comparable computational efficiency.
BibTeX
@inproceedings{wang2023language,
  author = {Wenxuan Wang and Guodong Ma and Yuke Li and Binbin Du},
  title = {{Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition}},
  booktitle = {Interspeech},
  year = {2023},
  url = {https://openreview.net/forum?id=XhuU6LsCxP},
}
Figure