Abstract
This paper introduces a "one-stop" training framework designed to simultaneously optimize high-capacity (e.g., Mixture-of-Experts) and low-capacity (e.g., dense) models. Traditional Knowledge Distillation (KD) is often slow and unidirectional. The proposed Two-Stage Joint-Training (TSJT) algorithm enables bidirectional learning through a shared objective, followed by independent fine-tuning. Experiments on the WMT10 benchmark demonstrate that this approach not only accelerates convergence but also yields significant performance improvements for both large and small models compared to serial training methods.
Introduction
In the era of Large Language Models (LLMs), researchers face a dilemma: high-capacity models like Mixture-of-Experts (MoE) offer state-of-the-art performance but are too computationally expensive for deployment on edge devices. Conversely, low-capacity models are efficient but often lag in accuracy. The standard solution, Knowledge Distillation (KD), involves training a "student" model to mimic a pre-trained "teacher." However, KD is inherently slow due to its serial nature and unidirectional, as information only flows from the teacher to the student.
This work addresses these limitations by proposing a framework where models of different capacities are trained together from scratch in a single run. This "one-stop" training approach allows for bidirectional learning, where both models can benefit from each other's optimization process.
Methods
The authors introduce two core components to facilitate multi-capacity training:
Composite Model Architectures
To accommodate different deployment needs, two architectural strategies are proposed:
- Shared Architecture: A large MoE model and a smaller dense model share specific transformer layers. This forces a common representation but restricts the size ratio between the models.
- Independent Architecture: The models are entirely separate in terms of parameters but are trained with a joint objective. This allows for a wider range of capacities, such as pairing a massive MoE model with a tiny device-sized model.
Two-Stage Joint-Training (TSJT)
The "secret sauce" of the framework is the TSJT algorithm, which divides training into two distinct phases:
- Stage 1 (Collaborative): Models are trained with a joint objective. In addition to standard cross-entropy loss, a Kullback-Leibler (KL) divergence loss is added to force consistency between models, allowing for mutual supervision.
- Stage 2 (Independent): Once models reach an optimal region—defined by a KL threshold ($t_{sep}$)—the consistency constraint is removed. Models then perform local fine-tuning to reach their individual optima.
Results and Discussion
The framework was evaluated on the WMT10 multilingual machine translation benchmark, comprising 32.5 million sentence pairs across 10 language directions.
Performance Gains
On the WMT10 benchmark, TSJT-trained MoE models outperformed standard MoE baselines by 1.4% to 4.1% BLEU. Smaller dense models (320M parameters) also saw an average improvement of 1.3% in X→En translation. These results suggest that the joint training signal acts as a powerful regularizer, helping both models find better solutions.
Efficiency and Convergence
Analysis of optimization trajectories revealed that TSJT models converge significantly faster than models trained individually. By sharing the learning process, the models reach lower loss levels in fewer updates, potentially saving substantial GPU resources when training a family of models.
Conclusion
The "one-stop" training framework effectively bridges the gap between high-performance large models and efficient small models. By leveraging bidirectional learning and a two-stage optimization process, it achieves superior performance and efficiency compared to traditional Knowledge Distillation. Future work will focus on extending this method to encoder-only and decoder-only architectures and exploring its application to fine-tuning pre-trained models.
Summary updated