The central promise of Mixture-of-Experts (MoE) is a "free lunch" in neural network scaling: the ability to increase a model's total parameter count by orders of magnitude without a proportional increase in the computational cost per token. By using a gating mechanism to route inputs to only a small subset of "experts," these models achieve sparse activation.
However, research across domains—from language modeling and speech recognition to computer vision—reveals a consistent pattern: the benefits of adding more experts are not infinite. Instead, MoE architectures face a "seesaw" problem where improving specialized performance often risks degrading general knowledge or hitting a hard ceiling of diminishing returns.
The Specialization-Generalization Seesaw
The primary goal of an MoE system is to allow sub-networks to master specific niches. In long-tailed recognition, Jiarui Cai and colleagues describe this in ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot as a "seesaw" problem. When a model becomes too specialized in rare "tail" classes, it often loses its grasp on common "head" classes.
This tension exists because specialization requires a delicate balance. If experts are too distinct, they fail to share general knowledge; if they are too similar, the benefits of sparsity vanish. Jin Ma and Siyuan Qiao address this in Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss by introducing the Expert-Router Coupling (ERC) loss. This forces routers to "understand" expert capabilities, ensuring that specialization is intentional rather than accidental.
The Law of Diminishing Experts
A recurring theme across the literature is that the marginal utility of adding experts follows a power law that eventually saturates. Aidan Clark and the team at DeepMind formalized this in Unified Scaling Laws for Routed Language Models, showing that as the base model size increases, the relative gain from adding more experts actually decreases.
Why does this saturation occur? Several mechanisms have been identified:
- Signal Dilution: As the number of experts grows, each individual expert sees fewer tokens during training. Shruti Bhosale and Angela Fan observed in their WMT21 News Translation Task Submission that if task diversity is low, experts receive insufficient exposure to develop effective representations.
- The Distraction Effect: In RouterRetriever, Hyunji Lee and Kyle Lo explain that a larger pool of experts makes the routing task itself more difficult. The probability of the router selecting a sub-optimal expert increases, creating a performance plateau even as the theoretical capacity of the expert pool grows.
- Routing Fluctuation: Xing W argues in MaskMoE that dynamic routing can be unstable. If a token doesn't consistently reach the same expert, the model fails to build deep, specialized knowledge for that token type.
Optimizing Expert Granularity
If adding more experts isn't always the answer, the focus shifts to the granularity and organization of those experts. Xiaoye Qu and Yu Cheng explored this in LLaMA-MoE v2, finding that while moving from coarse to medium-grained experts helps, partitioning the model into extremely fine-grained experts eventually degrades performance.
To combat the inefficiencies of massive expert pools, researchers are moving toward structured collaboration. In Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy, Mohan Zhang suggests forcing experts into "squads." By constraining which experts can work together, the system reduces communication overhead and improves accuracy through more focused specialization. Similarly, Wan‐Wan Lin and Wenqiao Zhang introduced TeamLoRA, which uses a game-theoretic competition module to optimize how experts are selected, ensuring they complement rather than redundantize each other.
Beyond Language: Universal Sparsity
The principles of MoE scaling extend far beyond text. In computer vision, Tianlong Chen 0001 developed AdaMV-MoE, which adaptively scales the number of activated experts based on the complexity of the visual task. In time-series forecasting, Ziqi Liu 0001 showed in FreqMoE that experts can specialize in specific frequency bands, though even here, dividing the spectrum too finely leads to "frequency range fragmentation" and performance loss.
Ultimately, the research suggests that MoE is not just a way to build "bigger" models, but a way to build more "organized" ones. As Longfei Yun and Yao Fu conclude in Toward Inference-optimal Mixture-of-Expert Large Language Models, the goal is to find the "inference-optimal" configuration—where the number of experts is balanced against the model's ability to actually route to them effectively.