Mitigating Expert Collapse in Sparse Mixture of Experts

1,281 concepts776 papers33 proposals

Sparsely activated Mixture-of-Experts (MoE) models offer a tantalizing promise: the ability to scale a neural network's parameter count by orders of magnitude without a corresponding increase in computational cost. Instead of passing an input token through a massive, dense feed-forward network, a gating mechanism—or router—dispatches each token to a small subset of specialized sub-networks, known as "experts."

However, when trained out of the box, this architecture suffers from a severe pathology: expert collapse. Left to its own devices, the learned routing mechanism naturally degrades into a state where a tiny fraction of available experts are overloaded with the vast majority of tokens, while the remaining experts sit idle or become entirely redundant. Understanding why this collapse happens, and how to prevent it, is essential to unlocking the true capacity of sparse models.

The Intuition Behind Routing Collapse

To understand why routers collapse, consider the state of an MoE model at the beginning of training. The router's weights are initialized randomly, meaning it dispatches tokens to experts essentially at random.

[Input Tokens] ---> [ Learned Router ] ---> [Expert 1] (Overloaded / "Hot")
                                      ---> [Expert 2] (Dead / "Cold")
                                      ---> [Expert 3] (Dead / "Cold")

By pure statistical chance, one expert will receive slightly more tokens than the others, or it will happen to receive tokens that yield a cleaner gradient signal. As a result, this preferred expert updates its weights faster and becomes slightly more competent than its peers. The router quickly notices this competence and adjusts its gating logic to send even more tokens to this "favorite" expert.

This triggers a runaway, self-reinforcing feedback loop. The favored expert gets heavily trained and optimized, making it even more attractive to the router, while the neglected experts receive no data, starve, and become "dead experts." As noted by Carlos Riquelme et al. in Scaling Vision with Sparse Mixture of Experts, managing this collapse is the central challenge of training sparse architectures.

The Dual Failure Modes: Starvation vs. Homogenization

The consequences of expert collapse manifest as a delicate tension between hardware efficiency and representation quality.

1. Hardware Load Imbalance

When the router funnels the majority of tokens to a handful of "hot" experts, the underlying distributed hardware suffers. In large-scale deployments, different experts are typically sharded across different physical GPUs. As Mike Lewis et al. demonstrate in BASE Layers: Simplifying Training of Large, Sparse Models, an uneven token distribution leads to severe hardware bottlenecks. The GPUs hosting the overloaded experts suffer from memory overflows and execution stalls, while the remaining GPUs sit idle, wasting massive amounts of available compute resource.

2. Expert Homogenization

To prevent hardware starvation, early MoE designs introduced aggressive auxiliary load-balancing losses during training. These losses penalize the router if it deviates from a perfectly uniform distribution of tokens across experts.

However, this introduces a secondary failure mode. If the balancing constraint is too rigid, the router is forced to distribute tokens indiscriminately to satisfy the loss function. As Hongcan Guo et al. argue in Advancing Expert Specialization for Better MoE, forcing every expert to handle a wide variety of disparate tokens prevents them from developing specialized functional niches. Instead, the experts learn nearly identical representations—a phenomenon termed "expert homogenization" by Boan Liu et al. in Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer. The model's effective parameter capacity collapses back down to that of a single dense layer, defeating the purpose of the MoE architecture.

Rethinking the Router: Algorithmic Solutions

To break out of the trap of starvation and homogenization, researchers have developed alternative routing paradigms that guarantee balance by design rather than relying on finicky auxiliary losses.

Inverting the Choice: Expert Choice Routing

Standard MoE architectures rely on Token Choice routing, where each token independently selects its top-$k$ preferred experts. Because tokens act independently, they can easily bottleneck on a single popular expert.

Token Choice:  [Token A] ---> Chooses ---> [Expert 1] (Bottleneck)
               [Token B] ---> Chooses ---> [Expert 1]

Expert Choice: [Expert 1] ---> Chooses ---> [Token A, Token C] (Fixed Capacity)
               [Expert 2] ---> Chooses ---> [Token B, Token D] (Fixed Capacity)

To invert this dynamic, Tianlin Liu et al. in Routers in Vision Mixture of Experts: An Empirical Study explore Expert Choice routing. Instead of tokens choosing experts, each expert is allocated a fixed capacity and chooses the top-$k$ tokens it is most qualified to process. Because every expert is forced to select a full batch of tokens, perfect load balancing is algorithmically guaranteed. This eliminates the need for auxiliary balancing losses entirely, allowing the experts to specialize deeply without interfering with the primary training objective, a benefit also leveraged by Haotian Sun et al. in EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing.

Routing as Linear Assignment

Another approach is to frame token routing as a classic constrained optimization problem. In BASE Layers, the authors formulate token allocation as a linear assignment problem. By solving this globally for an entire batch of tokens, the model mathematically guarantees that every expert receives an identical number of tokens, ensuring optimal hardware utilization while allowing the model to maximize its parameter capacity.

Deterministic Hashing

Alternatively, one can bypass learned routing entirely to eliminate training instability. The authors of Hash Layers For Large Sparse Models use a fixed, parameter-free hash function of the input token ID to determine its expert destination. While this completely prevents learned collapse, it introduces a different vulnerability: if the natural language data follows a highly skewed Zipfian distribution (where a few words appear vastly more often than others), certain experts will still face hardware utilization imbalances due to the underlying data distribution.

System-Level Adaptations and Post-Training Recovery

When algorithmic adjustments during training are not enough, the underlying hardware infrastructure or post-training compression can mitigate the effects of collapse.

Dynamic Device Placement

If token routing is inherently unpredictable and data-dependent, the hardware layout can be adapted dynamically to match the workload. Frameworks like FlexMoE, developed by Xiaonan Nie et al. in FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement, monitor data traffic in real time. Instead of keeping experts statically bound to specific GPUs, the system dynamically duplicates and migrates "hot" experts across physical devices to balance the computational load on the fly.

Exploiting Redundancy Post-Training

When a model finishes training with a degree of expert collapse or homogenization, it leaves behind a trail of underutilized or redundant parameters. Rather than wasting memory and compute on these modules during deployment, post-training techniques can reclaim efficiency.

As demonstrated in Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models, permanently pruning or dynamically skipping redundant experts can halve the memory footprint of an MoE model during inference with negligible impact on accuracy. Similarly, Lei Liu in Enhanced Expert Merging for Mixture-of-Experts in Graph Foundation Models shows that overlapping, homogenized experts can be mathematically compressed and merged into a single weight matrix. This achieves the streamlined efficiency of a dense model while preserving the collective knowledge base of the multi-expert ensemble.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_expert_collapse_a2b7bc65,
  title = {expert collapse},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-collapse/rdr_a2b7bc65645e4672822765e6ce98d6cf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_expert_collapse_a2b7bc65_v1,
  title = {expert collapse},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-collapse/rdr_a2b7bc65645e4672822765e6ce98d6cf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake