TL;DR
The researchers at XPeng Inc. have developed a framework to consolidate multiple domain-specialist LLMs into a single, high-performance dynamic Mixture-of-Experts (MoE) model. By introducing Calibrated Multi-teacher On-Policy Distillation (C-MOPD) and an optimized execution path, they achieved near-perfect integration of specialist capabilities while speeding up the training process by 2.86×.
What Problem Does This Paper Address?
In modern LLM post-training, it is common to develop "specialists"—models fine-tuned via Reinforcement Learning (RL) to excel in specific domains like coding or mathematics. However, deploying a fleet of separate specialists is computationally expensive and difficult to manage.
The challenge lies in capability integration: how do you merge these specialists into one student model without losing their unique strengths? Previous methods often suffered from "exposure mismatch" (where the student sees data differently than the teacher did) or "loss scale imbalance" (where one teacher's loss function dominates others during training). Furthermore, training dynamic MoE models is notoriously slow due to the overhead of routing tokens to different experts and the serial nature of generating data (rollouts) before updating the model.
Methods
The authors propose a three-stage pipeline: Shared SFT $\rightarrow$ Domain-Specific RL $\rightarrow$ C-MOPD Consolidation.
1. Calibrated Multi-teacher On-Policy Distillation (C-MOPD)
To merge specialists, the student model generates its own responses (on-policy), and a task router selects the appropriate specialist to act as a teacher. The student then tries to match the teacher's probability distribution for the top tokens.
- Loss-Scale Calibration: Since a "coding" loss might have a different magnitude than a "math" loss, the authors apply teacher-wise normalization.
- Token-Level Clipping: This prevents outlier tokens from producing massive gradients that could destabilize training.
2. Efficient Dynamic-MoE Execution
To solve the training bottleneck, they implemented several systems-level optimizations:
- RW32 (Bounded-width dispatch): Standard MoE models reserve many "slots" for experts. By capping the dispatch width to 32 experts per token, they reduced the memory footprint by 8× with negligible impact on accuracy.
- Final-Route Replay (R3): This ensures that the expert routing chosen during the data generation phase is exactly reused during the training update phase, maintaining consistency.
- AsyncOverlap: They overlapped the "rollout" (generation) and "actor" (training) phases. While the trainer updates the model using batch $i$, the rollout engine generates batch $i+1$ using a slightly older version of the policy (one-step policy lag).
Datasets
- SFT Stage: 5.27M examples covering math, code, tool use, and dialogue.
- Consolidation Pool: 51,500 high-quality prompts specifically curated for Math, Code, Instruction Following (IF), and Agent tasks.
- Evaluation: A massive suite including MMLU, CMMLU, C-Eval, GSM8K, MATH, HumanEval, MBPP, IFEval, and BFCL v4 (Berkeley Function-Calling Leaderboard).
Key Insights and Ideas
- Calibration is the Secret Sauce: The "aha moment" here is that raw distillation losses are not comparable across domains. Without normalization and clipping, the student model's performance significantly degrades because it over-optimizes for the "loudest" teacher.
- MoE Efficiency is a Systems Problem: The 2.86× speedup wasn't achieved through a better optimizer, but through clever scheduling (AsyncOverlap) and reducing redundant expert slots (RW32).
- Dual-Mode Support: The model (Turing-20B-A2B) natively supports both "Think" mode (Chain-of-Thought reasoning) and direct response, controlled by simple prefix markers like
<think>.
Key Results
- Integration Success: The consolidated model achieved a normalized integration score of 0.9988, meaning it successfully captured nearly 100% of the gains made by the individual specialists.
- State-of-the-Art Reasoning: In reasoning-enabled benchmarks (with an 8K response cap), Turing-20B-A2B outperformed Qwen3-8B on all four major math and code benchmarks, achieving an unweighted mean of 85.92 vs. Qwen's 80.33.
- Training Speed: The optimized execution path reduced the time per training step from 398.7 seconds to 139.3 seconds.
Limitations and Future Work
- GPQA Performance: The authors noted a slight decrease (5.05 points) in the GPQA Diamond benchmark compared to the SFT baseline, suggesting that while reasoning improved, some niche scientific knowledge might have been "washed out" during consolidation.
- Policy Lag: While AsyncOverlap provides a massive speedup, it introduces a "one-step policy lag." While this didn't hurt performance in this study, the long-term stability of this approach for even larger models or longer training runs remains an open question.
- Future Directions: The framework currently uses fixed teacher weights. Future work could explore dynamic weighting where the student learns which teacher to trust more based on real-time performance metrics.
Summary updated