In a standard Mixture-of-Experts (MoE) architecture, a gating network (or router) and a set of specialized sub-networks (experts) are trained simultaneously using an end-to-end objective. While elegant in theory, this joint optimization introduces a severe chicken-and-egg dilemma known as co-adaptation.
The router cannot make optimal routing decisions because the experts have not yet developed specialized capabilities. Conversely, the experts cannot specialize because the router's assignments fluctuate wildly from one training step to the next. As Damai Dai et al. observe in StableMoE: Stable Routing Strategy for Mixture of Experts, this "routing fluctuation" severely degrades sample efficiency. Furthermore, early routing errors can permanently corrupt local expert representations before a stable global boundary structure is even established, a phenomenon highlighted by Marc Toussaint et al. in Learning discontinuities with products-of-sigmoids for switching between local models.
To resolve this optimization conflict, a powerful paradigm has emerged: decoupled expert-router training. By breaking the dependency loop and separating the optimization of the selection logic from the underlying representations, researchers can achieve stable, highly specialized modular architectures.
Expert-First Specialization
One prominent strategy for decoupling is to isolate and train the experts on specific data distributions, features, or domains before introducing or updating the routing logic. This ensures that each expert achieves mastery over a well-defined subspace without being distracted by data from other domains.
For instance, in monocular depth estimation, Songsong Yu et al. developed a framework in DME: Unveiling the Bias for Better Generalized Monocular Depth Estimation that first optimizes experts on restricted distance brackets. Once these experts are frozen, a routing network is trained to stitch their predictions together. Similarly, in diffusion models, Shaobin Zhuang et al. show in TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision that training individual LoRA experts on isolated noise intervals before learning a routing mechanism allows the experts to build deep, specialized representations.
This concept extends naturally to large language models (LLMs) and multi-task learning. Junmo Kang et al. in Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts train specialized modules independently and keep them frozen while a routing layer learns to dynamically combine them. By isolating the initial training phase—whether by data features like seasonal and trend components in time-series forecasting (Donghao Luo et al. in DecompNet: Enhancing Time Series Forecasting Models with Implicit Decomposition) or by distinct semantic clusters in vision-language models (Luong Tran et al. in CLIP-FMoE: Scalable CLIP via Fused Mixture-of-Experts with Enforced Specialization)—each module maximizes its representational potential without being diluted by irrelevant data.
Router-First Stabilization
Alternatively, decoupling can happen in reverse: stabilizing the routing policy first so that experts are exposed to consistent data distributions from the very beginning of their training lifecycle.
In StableMoE, Damai Dai et al. first learn a routing strategy and distill it into a lightweight, fixed router before training the main expert parameters. This eliminates routing fluctuation entirely during the primary training phase.
A similar framing appears in "Learning to Defer" (L2D) contexts, where a system must decide whether to act or delegate to an external expert. Anqi Mao et al. demonstrate in Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer that treating predictive experts as fixed, pre-trained components allows the optimization process to focus exclusively on the delegation policy, providing strong theoretical guarantees for multi-expert routing.
Multi-Stage Warmups and Bi-Level Optimization
To capture the benefits of both independent specialization and global cohesion, many modern architectures employ multi-stage or bi-level training loops.
A common pattern is a warmup phase where experts or routers are trained independently, followed by a joint fine-tuning phase. For example, Ning Ding et al. show in Fusing Highly Specialized Language Models for Comprehensive Expertise that first training a gating network while keeping specialized models frozen, followed by joint fine-tuning, prevents catastrophic forgetting in the underlying specialists. Weikai Li et al. utilize a similar two-stage approach in Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis to prevent "expert polarization," where a single expert dominates early learning.
Weisi Liu demonstrates in Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts that a preliminary training phase ensures routing networks learn to correctly associate data with historical clusters before full model deployment.
Joint End-to-End Training (Fluctuation & Polarization):
[Input] ──> [Router (Unstable)] ──> [Experts (Unable to Specialize)]
Decoupled Training (Stage 1: Isolate & Freeze):
[Domain Data] ──> [Isolated Expert Training] ──> Freeze Expert Weights
Decoupled Training (Stage 2: Optimize Routing):
[Input] ──> [Router (Learns Clean Boundaries)] ──> [Frozen Experts]
In decentralized settings like federated learning, Dongyang Fan et al. propose a bi-level optimization strategy in On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists. Here, the routing logic is optimized on a separate validation set to ensure accurate selection between general and specialized knowledge, while the experts are updated using primary local data.
The Necessity of Re-Integration
While decoupling provides stability, completely isolating the components throughout the entire training lifecycle can introduce its own drawbacks. Early work by Edwin V. Bonilla et al. in Kernel Multi-task Learning using Task-specific Features and Alex Bewley et al. in Fine-Grained Classification via Mixture of Deep Convolutional Neural Networks demonstrated that purely independent training can lead to suboptimal global performance because the experts cannot adapt dynamically to the exact boundaries drawn by the router.
Therefore, the most effective modern implementations treat decoupling not as a permanent separation, but as a bootstrapping mechanism. By using decoupled training as an initialization or warmup phase—as seen in works ranging from simultaneous translation (Shaolei Zhang et al. in Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy) to cloud scheduling (Ya Shen et al. in Deft Scheduling of Dynamic Cloud Workflows with Varying Deadlines via Mixture-of-Experts)—the system achieves the best of both worlds: highly specialized, stable representations and globally optimized routing logic.
Key Foundational Work
Learning discontinuities with products-of-sigmoids for switching between local models (2005) — contributed 2 concept elements to this direction.
Kernel Multi-task Learning using Task-specific Features (2007) — contributed 1 concept elements to this direction.