Aurora: Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning

Rongsheng Wang, Haoming Chen, Ruizhe Zhou, Yaofei Duan, Kunyan Cai, Han Ma, Jiaxi Cui, Jian Li, Patrick Cheong-Iao Pang, Yapeng Wang, Tao TanarXiv 2023 · DOI · arXiv

Introduction

The Mixtral-8x7B model represents a significant milestone in open-source AI, utilizing a Sparse Mixture-of-Experts (MoE) architecture to achieve high performance with the computational footprint of a much smaller model. However, its native support for Chinese tasks has historically been a weak point. When prompted in Chinese, the original model frequently exhibits hallucinations, language mixing (often reverting to English or German), and disjointed reasoning.

To address these limitations, researchers have introduced Aurora, a version of Mixtral-8x7B specifically instruction-tuned to excel in Chinese conversation. This project explores whether standard instruction-tuning techniques, typically applied to dense models, can effectively "activate" specific linguistic capabilities within a sparse MoE framework.

The Aurora Pipeline

Aurora was developed using a supervised fine-tuning (SFT) approach designed for hardware efficiency. The researchers utilized mixtral-8x7b-instruct-v0.1 as the base, which activates only 2 out of 8 experts per token. To manage the model's massive parameter count on accessible hardware, the team employed QLoRA (4-bit Quantized Low-Rank Adaptation). This method involves freezing the base model in 4-bit precision and training only small, low-rank adapter matrices.

The Aurora training pipeline, illustrating the transition from raw paired data to the final instruction-tuned model.

The training was conducted over 3 epochs on an NVIDIA H100 (80GB), maintaining a learning rate of 5e-5. This configuration allowed the entire training process to fit within approximately 45GiB of VRAM, demonstrating the accessibility of tuning large MoE models.

Data Composition and Distribution

The effectiveness of Aurora stems from a curated pipeline of approximately 176,678 dialogue rounds. The authors integrated and cleaned three primary Chinese instruction-following datasets:

  • alpaca_data_zh_51k: 51,000 Chinese-English pairs refined through machine translation and manual editing.
  • alpaca_gpt4_data_zh: 52,000 high-quality instructions generated by GPT-4.
  • sharegpt_70k: A general-domain multi-turn dialogue dataset.
Distribution of the 176K instruction-following examples across various subject domains, showing a heavy emphasis on Computer and Natural sciences.

The final dataset comprises 141,345 single-round and 35,333 multi-round conversations across 20 categories. As shown in the distribution, the highest concentrations are in computer science (43.7%), natural sciences (14.7%), and general science (13.7%).

Key Results and Insights

Aurora demonstrated dramatic improvements over the base Mixtral model and other open-source competitors across several key benchmarks:

  • C-Eval: Aurora scored 51.9, significantly outperforming the base Mistral-7B-Instruct (39.2) and nearly matching ChatGPT (52.5).
  • CMMLU: The model reached 49.69, surpassing LLaMA-2-70B-Chat (43.3).
  • MMLU: It maintained a high score of 67.74, proving that the addition of Chinese capabilities did not result in "catastrophic forgetting" of its original knowledge base.

A significant insight from this work is that sparse MoE models are highly receptive to instruction tuning for language adaptation. The results suggest that the underlying logic learned during pre-training is language-agnostic and can be effectively re-mapped to new languages without needing to retrain the model's "router" mechanism from scratch.

Limitations and Future Work

While Aurora successfully handles complex Chinese tasks—such as writing ancient poetry or generating Python code—it is considered a preliminary release. The model can still be overly verbose or occasionally default to English structures during highly complex reasoning. Future work will explore Direct Preference Optimization (DPO) to better align the model with human safety and stylistic preferences, as well as expanding multi-turn dialogue capabilities.

Summary updated

BibTeX
@misc{wang2023aurora,
  author = {Rongsheng Wang and Haoming Chen and Ruizhe Zhou and Yaofei Duan and Kunyan Cai and Han Ma and Jiaxi Cui and Jian Li and Patrick Cheong-Iao Pang and Yapeng Wang and Tao Tan},
  title = {{Aurora: Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning}},
  year = {2023},
  eprint = {2312.14557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.14557},
}
Figure