Abstract
Mixtral, a representative sparse mixture of experts (SMoE) language model, has received significant attention due to its unique model design and superior performance. Based on Mixtral-8x7B-v0.1, in this paper, we propose Chinese-Mixtral and Chinese-Mixtral-Instruct with improved Chinese language abilities by adopting further pre-training and instruction fine-tuning. Experimental results show that our Chinese-Mixtral and Chinese-Mixtral-Instruct successfully improve Chinese understanding and generation performance while retaining the original English abilities. Then, we discuss several key questions when performing language adaptation on large language models, including the necessity of extending the language-specific vocabulary and the choice of the initialization model (foundation model v.s. instruction model), by providing empirical results and analysis. We also present the visualizations of each expert to examine their importance on downstream tasks.
Introduction
While Large Language Models (LLMs) like Mixtral-8x7B show incredible performance, they are predominantly trained on English data. Adapting these models to other languages (like Chinese) usually involves "continual pre-training" on target-language text. However, the community has largely relied on "best practices" developed for dense models like LLaMA—specifically, extending the vocabulary to include more native characters and starting from instruction-tuned versions. This study challenges these assumptions for Sparse Mixture of Experts (SMoE) architectures and introduces high-performing open-source Chinese Mixtral models.
Methodology
The authors developed two models: Chinese-Mixtral (base) and Chinese-Mixtral-Instruct.
Architecture
The models utilize the Mixtral-8x7B-v0.1 architecture. This design uses a "router" to select two out of eight experts for every token, effectively activating only 13B parameters during inference despite having a much larger total parameter count.
Training Strategy
The adaptation process followed two main stages:
- Continual Pre-training (CPT): Training the base Mixtral on 20GB of general Chinese corpus (~7 billion tokens).
- Supervised Fine-Tuning (SFT): Training the resulting model on 5 million Chinese instruction-following samples.
The researchers employed QLoRA (Quantized Low-Rank Adaptation) for efficiency, while performing full training on the embedding and LM head layers. Notably, they chose not to extend the original Mixtral tokenizer's vocabulary after finding it detrimental to performance.
Key Insights and Analysis
The study provides several critical findings regarding language adaptation:
- The Vocabulary Paradox: Traditionally, researchers add Chinese characters to the vocabulary to improve encoding efficiency. While this improved efficiency by ~32%, it caused a significant drop in downstream task performance. The original, smaller vocabulary appears sufficient, and the disruption to internal representations during extension may outweigh efficiency gains.
- Foundation vs. Instruct Initialization: A common question is whether to start adapting from a base model or one already instruction-tuned in English. The results clearly favor starting with the base model. Initializing from
Mixtral-Instructled to inferior Chinese performance, suggesting that English-specific instruction tuning "locks" the model into patterns that conflict with new language acquisition. - Expert Importance: Visualizing expert usage revealed that experts in lower layers are significantly more critical. Disabling a lower-layer expert caused massive performance drops, whereas disabling some upper-layer experts occasionally improved performance.
Experimental Results
The models were evaluated across several benchmarks, including C-Eval, CMMLU, and the Open LLM Leaderboard.
Performance and Human Preference
Chinese-Mixtral-Instruct outperformed the original Mixtral-Instruct and other 13B-class models on Chinese benchmarks. In the "Chatbot Arena," a human-evaluation platform, Chinese-Mixtral-Instruct achieved the highest Elo rating (1571), surpassing several larger established models.
English Preservation and Long Context
The model retained its English prowess, scoring competitively on the Open LLM Leaderboard (e.g., 71.53 on MMLU). Furthermore, despite being trained on a context length of only 1024, the models maintained the ability to handle up to 32K tokens and remained coherent even up to 128K tokens.
Conclusion
This work demonstrates that Mixtral-8x7B can be effectively adapted to Chinese while maintaining English capabilities. The findings suggest that traditional adaptation strategies, such as vocabulary extension and instruction-model initialization, should be reconsidered for SMoE architectures. Future work will explore Direct Preference Optimization (DPO) and deeper investigations into expert specialization.
Summary updated