Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model

Chong Li, Yingzhuo Deng, Jiajun Zhang, Chengqing Zong

Introduction

Multilingual Large Language Models (LLMs) face a fundamental trade-off known as the "curse of multilinguality." As models are scaled to support more languages within a fixed parameter budget, performance on individual languages often degrades due to parameter competition and negative transfer. This paper proposes Dynamic Mixture-of-Experts (DMoE), a "Group then Scale" framework designed to expand model capacity efficiently by specializing only the most language-sensitive layers.

Methodology: The DMoE Pipeline

The DMoE approach consists of three distinct phases designed to identify and address linguistic interference.

The DMoE pipeline: (a) Dynamic Language Clustering based on parameter deviation, (b) Dynamic MoE Layer Extension with specialized experts, and (c) Dynamic Language Adaptation for new languages.

1. Dynamic Language Clustering

Rather than relying on static linguistic taxonomies, DMoE uses a data-driven approach. The base model undergoes a brief "probing" phase—just 10 steps of fine-tuning on monolingual data for each target language. The resulting parameter deviation ($\Delta\theta$) serves as a "fingerprint." Languages exhibiting similar weight shifts are clustered together using a greedy algorithm, ensuring that experts are shared between languages that benefit from mutual positive transfer.

2. Dynamic MoE Layer Extension

The authors observe that language sensitivity is not uniform across model layers. Typically, layers near the input and output (embeddings and heads) show the highest deviation. DMoE converts only these "top-$\epsilon$" deviant layers into MoE layers. This selective scaling preserves a shared "concept space" in the middle layers while providing specialized capacity for language-specific surface features like grammar and syntax.

3. Specialized Training and Adaptation

To enforce expert specialization, a Language Group Classification Loss is introduced, guiding the MoE router to select the expert corresponding to the input token's language group. For adapting to new languages, DMoE identifies the most similar existing group, initializes a new expert from that group's weights, and fine-tunes only the new expert and router to prevent catastrophic forgetting of previous knowledge.

Experimental Results

The DMoE framework was evaluated across 128 languages using the BLOOM, Gemma, and Qwen2.5 model series, measured against benchmarks like XNLI and XWinograd.

Perplexity improvement (Δ PPL) across different language group configurations. Increasing the number of groups generally leads to greater performance gains relative to the pre-trained baseline.

Performance and Efficiency

DMoE achieved an average 11.4% improvement in perplexity compared to standard continual pre-training. Notably, it outperformed state-of-the-art branching models like X-ELM while requiring 3.6x fewer parameters on average. On reasoning tasks like XWinograd, DMoE showed a 1.6% improvement over the base model, suggesting that specialized experts better capture complex linguistic logic.

Robustness in Adaptation

When extending the model to four entirely new languages, the "Dynamic Language Adaptation" method limited performance regression on existing languages to just +0.7 PPL, significantly outperforming standard dense fine-tuning, which saw a +2.0 PPL increase. This demonstrates the framework's ability to integrate new knowledge without degrading existing capabilities.

Conclusion

Dynamic Mixture-of-Experts offers a scalable solution to the curse of multilinguality. By combining data-driven language grouping with selective layer specialization, DMoE achieves high performance and parameter efficiency. Future work may explore hybrid architectures that combine globally shared experts with language-specific ones to further optimize the balance between universal knowledge and linguistic nuance.

Summary updated

BibTeX
@inproceedings{li2025group,
  author = {Chong Li and Yingzhuo Deng and Jiajun Zhang and Chengqing Zong},
  title = {{Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure