Abstract
$\chi$FT (eXpert Fine-Tuning) is a novel training framework designed to enhance the performance of code Large Language Models (LLMs) by "upcycling" a dense model into a Mixture-of-Experts (MoE) architecture for instruction tuning, then merging those experts back into a single dense model. This approach achieves state-of-the-art results for small-scale models—such as a 67.1 pass@1 on HumanEval for a 1.3B parameter model—while maintaining the inference efficiency of the original dense architecture. By leveraging MoE capacity during training and reverting to a dense format for deployment, $\chi$FT effectively captures scaling benefits without the associated memory overhead.
Introduction
Instruction tuning is the critical process of fine-tuning pre-trained models to follow specific commands, a standard practice for creating coding assistants. While current research heavily emphasizes data quality and synthetic generation, the underlying training schemes have remained largely stagnant, typically relying on standard Supervised Fine-Tuning (SFT) on dense models. Sparse upcycling to MoE architectures offers a path to increased performance but is often hindered by high inference costs and the need for massive datasets. $\chi$FT addresses these challenges by utilizing MoE capacity only during the training phase, providing a "free lunch" in terms of performance and efficiency.
Methodology
The $\chi$FT framework consists of two primary stages: upcycling with a shared expert mechanism and learnable merging.
Shared Expert Upcycling
Unlike vanilla upcycling, $\chi$FT designates one expert in each layer as a "shared expert" that processes every token to capture general knowledge. A router assigns remaining tokens to specialized experts. To maintain stability during this transition, a routing weight normalization strategy is employed to ensure the output scale of the new MoE layer matches the original dense layer.
Learnable Merging
Following instruction tuning, the MoE model is "compiled" back into a single dense Feed-Forward Network (FFN). This is achieved through a learnable mechanism that determines optimal mixing coefficients for the experts. A fixed "shared expert rate" ($\lambda$) is used to protect the general knowledge captured during training, resulting in a model that retains its original inference size.
Experimental Setup
Models were trained on the evol-codealpaca-v1 dataset, comprising 110K instruction-output pairs. Evaluation was conducted across a diverse suite of benchmarks, including HumanEval(+) and MBPP(+) for Python, MultiPL-E for multilingual tasks, and DS-1000 for data science applications.
Results and Discussion
The application of $\chi$FT to the DeepSeek-Coder-1.3B base model resulted in a significant performance boost, achieving a 67.1 pass@1 on HumanEval—a 13% improvement over standard SFT.
Key Insights
- The Merging Advantage: Surprisingly, merging experts back into a dense model can improve performance over the MoE model itself, likely due to a form of learned ensembling.
- Role of Shared Experts: The shared expert acts as a stabilizer during the relatively short instruction tuning phase, preventing experts from becoming too specialized on limited data.
- Scalability: The framework scales effectively to larger models. A 6.7B parameter model reached 81.7 pass@1 on HumanEval, outperforming the MoE version while requiring 8x fewer parameters during inference.
Conclusion
$\chi$FT demonstrates that architectural upcycling followed by learned merging is a powerful, data-orthogonal strategy for instruction tuning. It achieves state-of-the-art performance for small code LLMs while maintaining optimal inference efficiency. Future work will explore automated hyperparameter tuning for the merging process and further investigate the theoretical foundations of why expert merging yields such robust performance gains.
Summary updated