Upcycling Dense Models into Sparse Mixture of Experts

39 concepts27 papers2 proposals

Scale is the primary driver of performance in modern deep learning. However, scaling up standard dense models quadratically increases computational cost (FLOPs) during both training and inference. Sparse Mixture-of-Experts (MoE) architectures offer an elegant alternative: they scale model capacity (parameters) while keeping the computational cost per token constant by activating only a small subset of "experts" for any given input.

Despite their efficiency, training MoE models from scratch is notoriously difficult. It suffers from severe routing instability—where the routing mechanism changes its decisions too rapidly for experts to specialize—and expert collapse, where a few experts are overloaded while the rest are ignored.

To bypass these challenges, a powerful paradigm has emerged: dense-to-sparse upcycling. Instead of starting from scratch, researchers initialize a sparse MoE model using the weights of a well-trained dense model. This provides a stable foundation of generalized knowledge, transforming a generalist dense model into a collection of specialized experts at a fraction of the computational cost.

The Core Mechanism: How to Split a Dense Layer

The central challenge of upcycling is determining how to map a single dense layer—typically the Feed-Forward Network (FFN) block of a Transformer—into multiple distinct experts. Researchers have developed three primary strategies to achieve this.

Weight Replication

The most straightforward approach is to copy the dense FFN weights exactly $N$ times to populate $N$ experts. As demonstrated by Aran Komatsuzaki et al., this preserves the functional capabilities of the original model immediately at initialization, completely avoiding the performance degradation associated with random initialization. This exact duplication strategy provides a warm start that has proven highly effective across modalities, including large vision-language models (Bin Lin et al.; Jiachen Li et al.) and text-image retrieval models (Xinze Wang et al.).

Weight Partitioning (MoEfication)

Instead of duplicating the entire layer, the dense layer's internal dimensions can be partitioned into smaller, independent sub-networks. Tong Zhu et al. and Kumari Nishu et al. show that slicing a dense FFN into finer-grained experts allows the model to dynamically allocate computation based on token difficulty. This post-training conversion allows simple tokens to bypass unnecessary computation while reserving full capacity for complex tokens. Ning Liao et al. extend this concept by partitioning experts across both intermediate and output dimensions, achieving massive throughput gains and latency reductions.

Importance-Based Sampling

Rather than exact duplication or rigid partitioning, weights can be sampled based on their historical activation magnitudes. Xingkui Zhu et al. demonstrate that sampling weights according to activation importance introduces early structural diversity among experts while ensuring that every expert retains high-quality, functional knowledge from the dense predecessor.

Why It Works: Solving the Chicken-and-Egg Problem

The fundamental reason dense-to-sparse initialization succeeds lies in separating representation learning from expert specialization.

When an MoE model is trained from scratch, the router and the experts face a chicken-and-egg problem: the router cannot learn where to send tokens because the experts produce meaningless outputs, and the experts cannot specialize because the router sends them inconsistent, rapidly changing mixtures of tokens. This synchronization failure causes severe training instability.

By starting with a pre-trained dense model, this problem is resolved. As noted by Xiaonan Nie et al. and Deyi Xiong et al., the dense weights provide the routing mechanism with stable, highly meaningful representations from the very first step of training. Dheeru Dua et al. observe that transitioning from a generalist dense architecture to a specialist sparse architecture results in higher expert entropy and more efficient parameter utilization. This confirms that sparsity is most effective when applied as a post-hoc specialization step to a model that has already mastered generalized features.

Overcoming the Pitfalls of Naive Upcycling

While upcycling drastically reduces the compute required to train sparse models, naive weight copying introduces unique optimization hurdles that require targeted solutions.

Breaking Symmetry to Avoid Plateaus

If all experts are initialized as identical copies, they receive identical gradients from symmetric routing, which can lead to a performance plateau where the model fails to differentiate its experts. To break this symmetry, Taishi Nakamura demonstrates that re-initializing a portion (e.g., 50%) of the weights forces the experts to diverge early in training, balancing knowledge retention with rapid specialization. Alternatively, Tingfeng Hui et al. suggest populating experts using diverse weight initializations drawn from a model's own intermediate training history, providing pre-specialized expert candidates from day one.

Aligning the Router

A randomly initialized router can disrupt pre-trained representations by sending tokens to arbitrary experts. To mitigate this, Wangyi Jiang et al. initialize routing weights using data-driven representations from the base model, providing a warm start that aligns the new routing logic with the model's pre-existing internal knowledge.

Mitigating Parameter Bloat

Replicating layers increases the total parameter count on disk, creating a massive memory footprint even if active FLOPs remain constant. Peng Ye et al. address this parameter explosion by treating experts as low-rank or sparse incremental "delta" matrices added to a single shared base layer, enabling extreme parameter efficiency during upcycling.

Broad Applicability Across Domains

The strategy of dense-to-sparse initialization is a robust scaling recipe that spans diverse architectures and modalities. Beyond standard language models, it has been successfully leveraged to build modular multilingual translation systems (Shangjie Li et al.), robust audiovisual speech recognition models that fuse video and audio without catastrophic forgetting (Yihan Wu et al.), and efficient speech recognition models using prob-sparse attention mechanisms (Lei Xie et al.). By recycling existing dense representations, the field has unlocked a computationally accessible pathway to scaling deep learning models.

Key Foundational Work

Dense-to-Sparse Gate for Mixture-of-Experts (2021) — contributed 2 concept elements to this direction.
Efficient Conformer with Prob-Sparse Attention Mechanism for End-to-End Speech Recognition (2021) — contributed 1 concept elements to this direction.
Network Morphism (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dense_to_sparse_initialization_58349e48,
  title = {dense-to-sparse initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dense-to-sparse-initialization/rdr_58349e4873044537a34e47bf5a43529b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dense_to_sparse_initialization_58349e48_v1,
  title = {dense-to-sparse initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dense-to-sparse-initialization/rdr_58349e4873044537a34e47bf5a43529b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake