UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition

Li‐Chen Fu, Shanyong Yu, Siqi Li, Lu Fan, Youzheng Wu, Xiaodong HeICASSP 2024

TL;DR

This paper introduces UME, a novel method to efficiently "upcycle" existing, smaller Automatic Speech Recognition (ASR) models into larger, more powerful Mixture-of-Experts (MoE) models. By reusing pretrained weights and employing smart training strategies like layer freezing and expert balancing, UME significantly reduces training time and computational costs while achieving state-of-the-art performance, outperforming models trained from scratch.

What Problem Does This Paper Address?

Training large-scale ASR models, like those with billions of parameters, has become a dominant trend for improving performance. However, this comes at a steep cost: immense computational resources, extensive training time, and complex hyperparameter tuning. Furthermore, simply scaling up dense models can lead to increased inference latency, making them less practical for real-time applications. The authors identify a gap: how can we leverage the power of larger models, specifically MoE architectures, without incurring the prohibitive costs of training from scratch, and while maintaining efficient inference? They propose "upcycling" existing, well-trained smaller models as a more efficient path to scaling.

Methods

The core of UME lies in its two-stage approach:

  • Upcycling Pretrained Dense Models into MoE: The authors take a pretrained dense ASR model and transform its Feed-Forward Network (FFN) layers into MoE layers. This involves creating multiple "expert" FFNs for each original FFN. Crucially, the weights from the original dense FFNs are copied to initialize these new experts. This ensures the upcycled MoE model starts with a strong, familiar foundation, mirroring the performance of the original dense model. A "top-k router" is used to selectively activate only a subset of experts for each input, keeping inference efficient. A key innovation here is a "renormalized sum" of the top-k expert outputs, which improves training stability compared to standard softmax-based routing.
  • Continued Training with Layer Freezing and Expert Balancing: To further enhance the upcycled MoE model, a specialized continued training strategy is employed.
  • Layer Freezing: Instead of fine-tuning the entire model, UME freezes most layers (down-sampling, attention, etc.) and only trains the newly introduced MoE layers (experts and router). This drastically reduces the number of trainable parameters, cutting down training time and mitigating "catastrophic forgetting" (where the model loses previously learned capabilities).
  • Expert Balancing: To ensure all experts are utilized effectively and prevent some from becoming over-trained while others remain underused, a differentiable load balancing loss is introduced. This loss encourages a more even distribution of data tokens across the experts within each MoE layer.

Datasets

The primary training dataset used is a large-scale Mandarin (90%) and English (10%) dataset comprising 170,000 hours of labeled audio. This dataset covers diverse domains, including internet content and recorded meetings.

For evaluation, the authors used a suite of public test sets:

  • SpeechIO (26 internet scenarios)
  • Wenetspeech test-net and test-meeting
  • Aishell-4 meeting
  • Gigaspeech dev and test (for English)

They also conducted extended experiments using a smaller, 10,000-hour dataset combining Wenetspeech and Aishell-4 to validate their approach on a different model architecture (Paraformer).

Key Insights and Ideas

  • "Upcycling" as an Efficient Scaling Strategy: The paper demonstrates that reusing existing, well-trained models is a highly effective and efficient way to achieve the benefits of larger, more complex architectures like MoE, bypassing the need for costly training from scratch.
  • Layer Freezing is Crucial for Mitigating Forgetting: By freezing most of the pretrained model's layers, UME effectively prevents the model from "forgetting" its strong initial performance while still allowing it to learn from the additional parameters in the MoE layers. This is a more direct approach than traditional knowledge distillation for this purpose.
  • Expert Balancing Enhances MoE Utilization: Simply adding more experts isn't enough; ensuring they are all actively contributing is key. The proposed expert balancing loss helps achieve this, leading to better overall performance and more efficient parameter usage.
  • MoE Performance Gains Without Prohibitive Latency: The use of sparsely-gated MoE layers, combined with the top-k routing mechanism, allows for a massive increase in model capacity without a corresponding significant increase in inference Real-Time Factor (RTF).

Key Results

  • Performance Improvement: UME achieved a relative error rate reduction of up to 11.9% compared to the pretrained dense baseline (Conformer-200M) on Mandarin/English datasets, while maintaining comparable RTF.
  • Training Time Reduction: UME reduced training time by up to 86.7% compared to training 1B-parameter models from scratch. Even when considering the initial pretraining time, the overall cost was significantly lower.
  • Ablation Studies: Layer freezing was shown to significantly mitigate performance degradation (forgetting) on challenging test sets, reducing CER by 6.9% on SpeechIO compared to not using it. Expert balancing consistently reduced CER and improved expert usage.
  • Paraformer Validation: The approach was successfully applied to the Paraformer model, achieving a relative CER reduction of 15.6% and demonstrating its generalizability across different ASR architectures.

Limitations and Future Work

  • Hyperparameter Tuning for Experts and Top-k: While the paper shows that increasing the number of experts and the top-k selection generally improves performance, they note that this needs to be balanced against computational costs. Future work could explore larger configurations.
  • Generalizability of Upcycling: The paper focuses on upcycling dense models into MoE. Further research could explore upcycling between different types of large models or investigate optimal strategies for different model architectures and pretraining objectives.
  • Code and Models: The authors state that their code and models will be released upon publication, which will be crucial for future research and reproducibility.

Summary updated

Abstract

Recent advancements in scaling up models have significantly improved performance in Automatic Speech Recognition (ASR) tasks. However, training large ASR models from scratch remains costly. To address this issue, we introduce UME, a novel method that efficiently Upcycles pretrained dense ASR checkpoints into larger Mixture-of-Experts (MoE) architectures. Initially, feed-forward networks are converted into MoE layers. By reusing the pretrained weights, we establish a robust foundation for the expanded model, significantly reducing optimization time. Then, layer freezing and expert balancing strategies are employed to continue training the model, further enhancing performance. Experiments on a mixture of 170k-hour Mandarin and English datasets show that UME: 1) surpasses the pretrained baseline by a margin of 11.9% relative error rate reduction while maintaining comparable latency; 2) reduces training time by up to 86.7% and achieves superior accuracy compared to training models of the same size from scratch.
BibTeX
@inproceedings{fu2024ume,
  author = {Li‐Chen Fu and Shanyong Yu and Siqi Li and Lu Fan and Youzheng Wu and Xiaodong He},
  title = {{UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
}
Figure