One-stop Training of Multiple Capacity Models

Lan Jiang, Haoyang Huang, Dongdong Zhang, Rui Jiang, Furu WeiarXiv 2023 · DOI · arXiv

Abstract

This paper introduces a "one-stop" training framework designed to simultaneously optimize high-capacity (e.g., Mixture-of-Experts) and low-capacity (e.g., dense) models. Traditional Knowledge Distillation (KD) is often slow and unidirectional. The proposed Two-Stage Joint-Training (TSJT) algorithm enables bidirectional learning through a shared objective, followed by independent fine-tuning. Experiments on the WMT10 benchmark demonstrate that this approach not only accelerates convergence but also yields significant performance improvements for both large and small models compared to serial training methods.

Introduction

In the era of Large Language Models (LLMs), researchers face a dilemma: high-capacity models like Mixture-of-Experts (MoE) offer state-of-the-art performance but are too computationally expensive for deployment on edge devices. Conversely, low-capacity models are efficient but often lag in accuracy. The standard solution, Knowledge Distillation (KD), involves training a "student" model to mimic a pre-trained "teacher." However, KD is inherently slow due to its serial nature and unidirectional, as information only flows from the teacher to the student.

This work addresses these limitations by proposing a framework where models of different capacities are trained together from scratch in a single run. This "one-stop" training approach allows for bidirectional learning, where both models can benefit from each other's optimization process.

Methods

The authors introduce two core components to facilitate multi-capacity training:

Composite Model Architectures

To accommodate different deployment needs, two architectural strategies are proposed:

  • Shared Architecture: A large MoE model and a smaller dense model share specific transformer layers. This forces a common representation but restricts the size ratio between the models.
  • Independent Architecture: The models are entirely separate in terms of parameters but are trained with a joint objective. This allows for a wider range of capacities, such as pairing a massive MoE model with a tiny device-sized model.

Two-Stage Joint-Training (TSJT)

The "secret sauce" of the framework is the TSJT algorithm, which divides training into two distinct phases:

  1. Stage 1 (Collaborative): Models are trained with a joint objective. In addition to standard cross-entropy loss, a Kullback-Leibler (KL) divergence loss is added to force consistency between models, allowing for mutual supervision.
  2. Stage 2 (Independent): Once models reach an optimal region—defined by a KL threshold ($t_{sep}$)—the consistency constraint is removed. Models then perform local fine-tuning to reach their individual optima.
The Two-Stage Joint-Training (TSJT) framework transitions from collaborative learning with KL loss in Stage I to independent fine-tuning in Stage II.

Results and Discussion

The framework was evaluated on the WMT10 multilingual machine translation benchmark, comprising 32.5 million sentence pairs across 10 language directions.

Performance Gains

On the WMT10 benchmark, TSJT-trained MoE models outperformed standard MoE baselines by 1.4% to 4.1% BLEU. Smaller dense models (320M parameters) also saw an average improvement of 1.3% in X→En translation. These results suggest that the joint training signal acts as a powerful regularizer, helping both models find better solutions.

Efficiency and Convergence

Analysis of optimization trajectories revealed that TSJT models converge significantly faster than models trained individually. By sharing the learning process, the models reach lower loss levels in fewer updates, potentially saving substantial GPU resources when training a family of models.

Cross-entropy loss trajectories demonstrate that TSJT-trained models (MoE and device) converge faster and achieve lower final loss compared to independent training baselines.

Conclusion

The "one-stop" training framework effectively bridges the gap between high-performance large models and efficient small models. By leveraging bidirectional learning and a two-stage optimization process, it achieves superior performance and efficiency compared to traditional Knowledge Distillation. Future work will focus on extending this method to encoder-only and decoder-only architectures and exploring its application to fine-tuning pre-trained models.

Summary updated

Abstract

Training models with varying capacities can be advantageous for deploying them in different scenarios. While high-capacity models offer better performance, low-capacity models require fewer computing resources for training and inference. In this work, we propose a novel one-stop training framework to jointly train high-capacity and low-capactiy models. This framework consists of two composite model architectures and a joint training algorithm called Two-Stage Joint-Training (TSJT). Unlike knowledge distillation, where multiple capacity models are trained from scratch separately, our approach integrates supervisions from different capacity models simultaneously, leading to faster and more efficient convergence. Extensive experiments on the multilingual machine translation benchmark WMT10 show that our method outperforms low-capacity baseline models and achieves comparable or better performance on high-capacity models. Notably, the analysis demonstrates that our method significantly influences the initial training process, leading to more efficient convergence and superior solutions.
BibTeX
@misc{jiang2023one,
  author = {Lan Jiang and Haoyang Huang and Dongdong Zhang and Rui Jiang and Furu Wei},
  title = {{One-stop Training of Multiple Capacity Models}},
  year = {2023},
  eprint = {2305.14066},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.14066},
}
Figure