Dynamic Expert Allocation for Mixture-of-Experts

This direction is split into other directions: static expert allocation. Its last version stays readable and citable here.

Version 1 of 172 papers96 conceptsfrozen 2026-10-04

In traditional Mixture-of-Experts (MoE) models, every token is treated as equally difficult. Whether a model is processing a simple comma or a rare technical term, it typically activates a fixed number of experts—usually two (Top-2 routing). This "one-size-fits-all" approach is computationally wasteful. It forces the model to spend excessive "brainpower" on trivial inputs while potentially starving complex ones of the parameters they need.

A growing body of research is shifting toward Dynamic Expert Allocation (or Dynamic-$k$ routing). The core intuition is simple: computational resources should be allocated based on the inherent complexity of the input. By allowing the number of active experts to vary per token, models can achieve higher accuracy with significantly lower average FLOPs.

From Fixed Top-k to Adaptive Thresholds

The fundamental limitation of standard MoE is the hard "Top-$k$" selection. This mechanism is non-differentiable and rigid. To move beyond it, researchers have introduced various thresholding strategies that allow the model to decide how many experts are "enough."

One common approach is Cumulative Probability Routing (or Top-$p$). Instead of picking $k$ experts, the model selects the minimum number of experts whose combined routing scores exceed a certain threshold. Yuanhang Yang and colleagues in XMoE demonstrate that this allows tokens to activate only the parameters they actually need, reducing computational load by over 50%. Similarly, Can Jin introduced DTop-p MoE, which uses a classical Proportional-Integral (PI) controller to tune these thresholds on the fly, ensuring the model stays within a specific computational budget while adapting to token difficulty.

Making Routing Differentiable

A major hurdle in training dynamic MoEs is that selecting a variable number of experts is typically a discrete, non-differentiable operation. Recent work has focused on making this process "smooth" so it can be trained end-to-end with standard gradient descent.

Jianfei Chen and Ziteng Wang proposed ReMoE, which replaces Top-$k$ with a simple ReLU-based gating mechanism. Because ReLU is differentiable, the model can naturally learn to output zero for unhelpful experts and positive values for useful ones. This allows the model to scale its compute budget token-by-token, mirroring the efficiency of Huffman coding in information theory—common tokens use few experts, while rare tokens trigger many.

Other researchers, such as Heng Zhao in ProbMoE, treat routing as a probabilistic inference problem. By using exact marginal gradients, they enable a "Dynamic-$k$" mode that prevents "expert collapse" (where only a few experts are ever trained) and allows the model to adaptively determine the optimal number of active parameters.

Learning the Optimal Brainpower

Rather than setting a manual threshold, some systems learn exactly how much effort to expend. Tongtian Yue and the team behind Ada-K Routing use a lightweight module to predict the optimal $k$ for each token. This approach slashed computational costs by 25% while actually improving accuracy, proving that forcing a model to use a second expert when the first is already confident can actually introduce noise.

Tao Lin took this further with DynMoE, which treats expert selection as a multi-label classification problem. By using per-expert trainable thresholds, the model automatically determines both the total number of experts in the system and how many to activate per input, eliminating the need for manual hyperparameter tuning.

Zero-Cost Sinks and Heterogeneous Experts

Innovation in dynamic allocation also extends to the architecture of the experts themselves.

  • The "Sink" Expert: Shen Yan introduced TC-MoE, which includes a parameter-free expert that always outputs zero. A standard Top-$k$ router can then "choose" to do nothing by routing a token to this zero-cost sink, effectively achieving dynamic sparsity within a static framework.
  • Heterogeneous Experts: HMoE breaks the tradition of identical experts. It uses experts of varying sizes and applies a "Parameter Penalty" loss. This forces the router to send simple tokens to tiny experts and reserve the massive, high-capacity experts for the most difficult parts of the data distribution.

Why It Works: The Intuition of Confidence

The success of these methods rests on the observation that routing distributions are often highly skewed. For many tokens, the gating network is extremely confident in a single expert. In these cases, adding a second expert (as in Top-2) adds marginal value but doubles the cost.

By monitoring the "confidence gap"—the difference between the top expert's score and its competitors—models like those proposed by Jiamin Li 0002 in Adaptive Gating can bypass redundant computation. This dynamic flexibility allows sparse models to finally match the robustness of dense models while maintaining the efficiency that makes MoE attractive in the first place.

Papers in this version

Showing 60 of 72 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_expert_allocation_for_mixture_of_edc16ada,
  title = {Dynamic Expert Allocation for Mixture-of-Experts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-expert-allocation-for-mixture-of-experts/rdr_edc16ada6f164ed7bdf5b9cf2ad99bac},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_expert_allocation_for_mixture_of_edc16ada_v1,
  title = {Dynamic Expert Allocation for Mixture-of-Experts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-expert-allocation-for-mixture-of-experts/rdr_edc16ada6f164ed7bdf5b9cf2ad99bac/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}