The core promise of Mixture-of-Experts (MoE) models is conditional computation: the ability to scale a model's capacity—its total parameter count—without a proportional increase in the computational cost per token. By routing inputs to only a sparse subset of specialized sub-networks ("experts"), MoEs achieve remarkable efficiency gains.
However, this architectural flexibility introduces a critical, often fragile hyperparameter bottleneck: expert count optimization. To build an effective MoE, designers must manually specify both the total number of available experts ($N$) and the exact number of active experts activated per input ($K$, typically via a Top-$K$ gating mechanism). Choosing these values is rarely a matter of "more is better." Instead, it is a highly sensitive balancing act where sub-optimal configurations can lead to severe training instability, parameter underutilization, or catastrophic performance degradation.
The Goldilocks Problem of Expert Capacity
Why can we not simply instantiate an arbitrarily large pool of experts and let the routing network sort it out? The fundamental issue is that an expert's utility is tightly bound to the distribution and volume of the underlying data.
Yating Liu demonstrates this constraint in DM-Adapter: Domain-Aware Mixture-of-Adapters for Text-Based Person Retrieval, showing that if the total number of experts exceeds the capacity supported by the training data, the routing mechanism begins to capture random noise rather than meaningful structural patterns. This destabilizes training and harms generalization. Conversely, if the expert count is too restrictive, the model suffers from capacity bottlenecks, forcing distinct data distributions to share the same parameters and causing negative interference.
Furthermore, the optimal configuration changes completely depending on the task and domain:
- Task-Specific Trade-offs: Xinze Wang et al. discovered in CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling that while expanding expert capacity yields significant accuracy gains for classification tasks, it can simultaneously degrade performance on retrieval tasks.
- Domain Dependence: In parameter-efficient fine-tuning, Zongqian Li et al. found in PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning that a fixed expert count is fundamentally sub-optimal when adapting models across diverse data distributions, as each domain requires a different degree of architectural sparsity.
This sensitivity is magnified in highly specialized fields. For instance, in molecular property prediction, Dongha Lee et al. note in Learning Topology-Specific Experts for Molecular Property Prediction that the number of experts must be manually tailored to the structural complexity of specific chemical datasets. Similarly, in single-cell multi-omics, Sukwon Yun et al. demonstrate in scMoE: single-cell Multi-Modal Multi-Task Learning via Sparse Mixture-of-Experts that the optimal expert count is highly sensitive to the exact granularity of input tokenization.
The Burden of Heuristic Tuning
Historically, finding the "Goldilocks" number of experts has relied on expensive, manual trial-and-error. This challenge is not new; Ronan Collobert et al. highlighted the severe sensitivity of modular systems to expert and gating configurations over two decades ago in A Parallel Mixture of SVMs for Very Large Scale Problems.
Despite the evolution from SVMs to deep Transformers, modern architectures still face the same lack of formal, automated rules. Whether optimizing multi-adapter systems (Hao Ban et al.), sequential recommendation models (Mingrui Liu et al.), or multi-modal image fusion networks (Yiming Sun et al.), researchers are frequently forced to run exhaustive grid searches to determine parameters like the number of sub-networks or the selection temperature (SeongKu Kang et al.).
This reliance on heuristics creates a major deployment bottleneck. As Pavel Rumiantsev et al. point out in Graph Knowledge Distillation to Mixture of Experts, a model cannot autonomously adapt its capacity to data of varying complexity if its underlying expert pool is rigidly predefined. If the environment changes—such as shifting market conditions in algorithmic trading (Xi Cheng et al.)—the static expert configuration quickly becomes obsolete.
From Static Hyperparameters to Learnable Sparsity
To overcome this brittleness, a major research direction focuses on replacing static, heuristic expert selection with automated and dynamic mechanisms. The goal is to allow the network to determine its own optimal expert count and routing paths on the fly.
Auto-Tuning via Multi-Label Routing
A prominent breakthrough in this direction is presented by Yongxin Guo et al. in Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models. Instead of forcing a rigid Top-$K$ selection, their framework, DynMoE, treats expert routing as a dynamic, multi-label classification problem.
Standard Top-K Routing (Static K):
Input Token ──> [ Router ] ──> Always selects exactly K experts (e.g., K=2)
Dynamic Multi-Label Routing (DynMoE):
Input Token ──> [ Router ] ──> Threshold-based activation ──> Activates 1, 3, or 0 experts
based on token complexity
By allowing the router to activate a variable number of experts based on the complexity of the incoming token, the model automatically optimizes both the total active parameter count and the expert utilization. This eliminates the need for grid searches and significantly lowers the active parameter overhead while maintaining competitive performance across vision and language tasks.
Explicit Expert-Router Coupling
Another reason static MoEs are highly sensitive to expert counts is that the routing network often operates blindly, assigning tokens without a clear understanding of what an expert has actually learned. Ang Lv et al. address this in Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss.
They introduce the Expert-Router Coupling (ERC) loss, which uses router embeddings as "proxy tokens" to force mathematical alignment between routing decisions and actual expert capabilities. By explicitly coupling the router's selection space to the experts' specialized knowledge, the system becomes far less sensitive to the total number of experts and the active token count, stabilizing performance across varying architectural scales.
Stabilizing Policy Updates
When MoEs are trained using Reinforcement Learning (RL), optimizing the expert count becomes even more volatile due to a phenomenon known as Router Shift, where routing paths change too drastically between policy updates. Di Zhang et al. tackle this in Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts. They demonstrate that trust thresholds are highly sensitive to the total number of experts. By introducing a soft-weighting mechanism called Router-Shift Policy Optimization (RSPO), they buffer the policy updates against changes in expert configuration, providing a smoother optimization landscape.
Conclusion
The transition from manually engineered, static expert pools to dynamic, self-optimizing MoE architectures represents a vital shift in scaling deep learning models. By treating expert counts and routing thresholds as learnable parameters rather than rigid hyperparameters, modern frameworks mitigate the risks of noise injection and capacity bottlenecks. This line of research ultimately paves the way for modular architectures that can autonomously adapt their computational footprint to the complexity of any given task or dataset.
Key Foundational Work
A Parallel Mixture of SVMs for Very Large Scale Problems (2002) — contributed 1 concept elements to this direction.
Global Gated Mixture of Second-order Pooling for Improving Deep Convolutional Neural Networks (2018) — contributed 1 concept elements to this direction.
Invocation-driven Neural Approximate Computing with a Multiclass-Classifier and Multiple Approximators (2018) — contributed 1 concept elements to this direction.