Mixture-of-Experts (MoE) architectures offer a compelling promise for large-scale deep learning: the ability to scale a model's parameter count—and thus its capacity—without a proportional increase in computational cost. By routing each token to only a sparse subset of specialized "expert" sub-networks, MoEs achieve high expressivity at a fraction of the active compute.
Yet, historically, realizing this promise has been notoriously difficult. Training an MoE model from scratch is plagued by severe optimization hurdles, data-hunger, and chronic training instability. To circumvent these "cold start" challenges, a powerful paradigm has emerged: sparse upcycling (or MoEfication). Instead of training sparse models from scratch, researchers initialize them from pre-trained dense checkpoints, transforming stable dense representations into highly efficient sparse networks.
The Root of Instability: The Co-Evolution Problem
To understand why upcycling works, we must first understand why training MoEs from scratch fails. When an MoE is initialized from scratch, it faces a chicken-and-egg problem involving two moving targets: the router (which decides where to send tokens) and the experts (which process the tokens).
At the start of training, the router is completely untrained and distributes tokens randomly. Because the routing decisions shift constantly, the data distribution seen by any individual expert is highly volatile. The experts cannot develop specialized skills because their inputs are a moving target; conversely, the router cannot learn an optimal routing policy because the experts have not yet developed distinct capabilities.
This chaotic co-evolution manifests as severe training pathologies:
- Loss Spikes and Oscillations: As observed by Wang et al., sparse architectures frequently encounter sudden loss spikes and oscillations along sharp directions of the loss landscape.
- Sequential Routing Collapse: Deeply stacked sparse layers exacerbate this issue. Xu et al. note that sequential routing through multiple sparse layers introduces cumulative instabilities, preventing models from effectively distributing features across deep networks.
- Exploitation of Spurious Signals: In complex optimization settings like Reinforcement Learning, the routing mechanism introduces additional layers of complexity. Dong et al. demonstrated that discrepancies in expert routing between the generation (rollout) and optimization phases create spurious optimization signals, leading to "objective-level hacking" and eventual model collapse.
These dynamics make training from scratch prohibitively expensive and fragile, often requiring complex auxiliary losses to prevent all tokens from being routed to a single expert (Wang et al.).
The Mechanics of Upcycling
Sparse upcycling bypasses the chaotic initialization phase by leveraging a fully trained, stable dense model as a foundation. The core recipe, popularized by Komatsuzaki et al., involves duplicating or partitioning the weights of a dense model to form the initial pool of experts.
Dense Layer (FFN) ───► Copy / Partition ───► [ Expert 1 ] [ Expert 2 ] [ Expert 3 ]
▲ ▲ ▲
└──────────┼──────────┘
[ Trained Router ]
Typically, the Feed-Forward Networks (FFNs) of the dense model are targeted for expansion. For example, Zhu et al. split the FFNs of a pre-trained LLaMA-2 model into multiple sparse experts, followed by a brief period of continual pre-training. This strategy is remarkably flexible; Zhang et al. demonstrated that both FFN and attention layers can be upcycled into experts, allowing the model to fully exploit specialized attention parameters rather than simply averaging them.
Rather than just copying weights blindly, recent work focuses on uncovering the latent specialization already present within dense models. The authors of DIVE into MoE discovered that pruning a dense model on different domain-specific datasets uncovers distinct, latent expertise within the same weights, providing a principled way to initialize highly diverse experts with minimal retraining.
Taming the Router
Even if the experts are initialized with high-quality dense weights, a randomly initialized router will still introduce chaotic token distribution. To prevent this, modern upcycling techniques focus on warm-starting the routers using the dense model's internal data representations.
If a router is initialized with random noise, the model experiences an immediate performance drop and optimization shock. To mitigate this, Jiang et al. showed that initializing routers based on the base model's internal representations—ensuring that tokens with similar activations are routed to the same experts from day one—dramatically improves training stability during instruction tuning.
A more formal approach to this problem is introduced by Bamba et al., who formulated the conversion of dense layers into experts as a Differentiable Optimal Transport (DOT) problem. By solving for an optimal assignment of dense neurons to sparse experts, the model jointly learns how to cluster neurons and route tokens to them smoothly, retaining 90% of the original dense model's performance before any continual training even begins.
Why Upcycling Works: The Intuition of Warm Routing
The fundamental reason upcycling works can be understood through the lens of representation safety.
In an MoE trained from scratch, a routing mistake is catastrophic: if a token is sent to the wrong expert, that expert outputs garbage gradients because it has no underlying capability to handle that token. This triggers the sharp oscillations and loss spikes identified by Wang et al..
In an upcycled MoE, the initial experts are derived from a fully functional dense model. This means that every expert initially possesses a baseline understanding of all tokens. If the router makes an imperfect routing decision early in the upcycling process, the destination expert can still process the token coherently. The dense weights act as a mathematical safety net.
As continual pre-training progresses, the router gradually refines its routing policy, and the experts slowly diverge to specialize in specific sub-domains. Because this specialization happens gradually from a stable baseline, the model avoids the chaotic feedback loops that destroy scratch-trained models.
This structural stability yields several profound practical advantages:
- Low-Data Viability: While scratch-trained MoEs are notoriously data-hungry, Xiong et al. demonstrated that the "dense-to-MoE" strategy allows sparse models to stabilize and thrive even in low-data regimes like literary translation.
- Numerical Robustness: MoE architectures are highly sensitive to lower-precision floating-point formats (such as 16-bit floats), which often trigger numerical divergence during routing. Lin et al. noted that upcycling strategies provide the necessary numerical stability to successfully train and deploy sparse models in these restricted precision environments.
- Cross-Modality Generalization: The stability of upcycling has enabled the expansion of MoEs into multimodal domains. Li et al. successfully applied "co-upcycling" to scale both the vision encoders and MLP connectors of multimodal LLMs, achieving massive capacity gains with negligible impact on inference costs.
By transforming the training process from a chaotic co-evolutionary struggle into a smooth, guided specialization of existing representations, upcycling has turned Mixture-of-Experts from a temperamental research curiosity into a scalable, production-ready architecture.
Key Foundational Work
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022) — contributed 1 concept elements to this direction.
TJUNLP: System Description for the WMT23 Literary Task in Chinese to English Translation Direction (2023) — contributed 1 concept elements to this direction.
Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity (2023) — contributed 1 concept elements to this direction.