Sparse Mixture-of-Experts (MoE) architectures offer a tantalizing promise: the ability to scale model capacity to billions or trillions of parameters without a proportional increase in computational cost. By activating only a small subset of "experts" for any given input, these models maintain the inference budget of a much smaller dense network.
However, training these models is notoriously difficult. Researchers have identified a recurring failure mode known as expert collapse (or router collapse), where the routing mechanism disproportionately favors a few experts while leaving others idle. This leads to a "rich-get-richer" cycle that undermines the very goal of MoE: specialization.
The Mechanics of Routing Failure
In a standard MoE, a learned router assigns tokens to experts. If an expert is chosen slightly more often early in training, it receives more gradient updates, becomes more "competent," and thus becomes even more likely to be chosen by the router.
As noted by Xiaonan Nie and colleagues, randomly initialized gates often exhibit this bias immediately, causing the model to waste its massive parameter count. This results in two distinct but related problems:
- Expert Collapse: A subset of experts is never used, effectively leaving parameters untrained.
- Expert Homogenization: Even when multiple experts are used, they may learn nearly identical representations. Boan Liu and Li Shen argue that this redundancy reduces the ensemble's expressive power to that of a single-expert model.
Diversifying the Expert Pool
To break the cycle of homogenization, researchers have introduced constraints that force experts to specialize. Boan Liu proposed OMoE (Orthogonal Mixture-of-Experts), which uses an orthogonal optimizer to ensure that experts update their parameters in different directions. By preventing experts from "drifting" toward the same local minima, the model preserves its representational diversity.
Another approach is to inject randomness into the selection process. Simiao Zuo and the team behind THOR (Transformer witH StOchastic ExpeRts) found that abandoning complex gating in favor of random expert selection—combined with consistency regularization—can actually outperform learned routers. This suggests that the primary value of MoE may lie in the sheer volume of parameters rather than the precision of the routing itself.
Solving the "Blind Spot" in Sparse Training
A fundamental issue with sparse training is that the router only receives feedback for the experts it selects. If an expert is not picked, the router never learns how useful that expert could have been.
Ashwinee Panda and Vatsal Baherwani addressed this with Default MoE. Their method uses an exponential moving average of past expert outputs as a "default" signal for unselected experts. This provides a dense gradient update to the router, allowing it to see the relative utility of the entire expert pool without the cost of running every expert.
Similarly, Nhuan Le Duc introduced Selective Sinkhorn Routing (SSR). By formulating token assignment as an optimal transport problem, SSR ensures balanced utilization. Crucially, they found that applying this computationally expensive "balanced" routing to only 0.1% of training steps is sufficient to prevent collapse while maintaining efficiency.
Structural Evolution and Merging
Rather than forcing sparsity from the start, some researchers suggest an evolutionary approach. Xiaonan Nie developed EvoMoE, which starts as a single dense expert and gradually splits into a sparse MoE. This "Dense-to-Sparse" transition allows the experts to mature before the routing becomes competitive.
In post-training scenarios, the focus shifts to managing the redundancy that already exists. Pingzhi Li and Yu Cheng introduced MC-SMoE, a framework that analyzes routing statistics to identify and merge redundant experts. By aligning and combining experts that the router treats similarly, they can reduce memory overhead by up to 80% with minimal performance loss.
Beyond Global Routing
Recent work also explores more structured routing to ensure domain-specific expertise. Jinguo Zhu proposed Attribute Routing in Uni-Perceiver-MoE to prevent "task interference," where different tasks (like vision vs. language) conflict over shared parameters. By explicitly conditioning the router on task attributes, the model ensures that experts develop the right kind of specialization for diverse data distributions.
This move toward structured sparsity is also seen in fine-tuning. Hanqing Zeng and Yinglong Xia developed S’MoRE, which organizes low-rank experts into a tree structure. This increases the number of possible routing combinations exponentially, allowing for high-capacity fine-tuning without the typical overhead of large-scale MoE deployment.