Dynamic Expert Allocation in Sparse Transformers

This direction is split into other directions: static expert allocation. Its last version stays readable and citable here.

Version 1 of 170 papers75 conceptsfrozen 2026-10-04

In a standard Mixture-of-Experts (MoE) model, every token is treated as an equal-opportunity consumer of computation. Whether the model is processing a simple comma or a complex technical term, the routing mechanism typically selects a fixed number of experts—usually two—to process the input. This "one-size-fits-all" approach, while easy to implement, ignores a fundamental truth of language and data: some inputs are much harder to process than others.

Recent research has shifted toward Dynamic Expert Allocation, a paradigm where the model adaptively decides how much "brainpower" to spend on each specific input. By breaking the rigid constraints of traditional Top-$K$ routing, these models achieve higher efficiency and better performance by focusing their parameters where they are needed most.

The Inefficiency of Fixed Routing

The core problem with traditional MoE models is the mismatch between token complexity and computational budget. As Jiamin Li and colleagues observe in Adaptive Gating in Mixture-of-Experts based Language Models, over half of all tokens in a typical sequence show a strong bias toward a single expert. Forcing these tokens to activate a second expert is essentially a waste of FLOPs.

Conversely, Quzhe Huang argues in Harder Tasks Need More Experts that difficult logical deductions or rare semantic concepts may actually be underserved by a fixed budget of two experts. When the computational resource is static, the model over-allocates to the "easy" tokens and under-allocates to the "hard" ones.

Mechanisms for Variable Capacity

To solve this, researchers have proposed several ways to make the number of active experts ($K$) a dynamic variable rather than a hyperparameter.

1. Cumulative Probability (Top-P)

Inspired by nucleus sampling in text generation, Can Jin and others introduced DTop-p MoE. Instead of picking the top $K$ experts, the router selects the smallest set of experts whose combined routing scores exceed a threshold $p$. If the router is very confident in one expert, it only uses one. If it is uncertain, it recruits more. To keep this within a hardware-friendly budget, they use a Proportional-Integral (PI) controller from classical control theory to adjust the threshold $p$ in real-time during training.

2. Entropy and Uncertainty

Another approach uses the "confidence" of the router to determine the budget. Mingjie Tang in DynMoLE uses Tsallis entropy to measure the uncertainty of the routing distribution. High entropy suggests the token is ambiguous or complex, triggering the activation of more experts. Similarly, Heng Zhao treats routing as a probabilistic inference problem in ProbMoE, allowing for a "Dynamic-K" mode that prevents "expert collapse" while adapting to token density.

3. Threshold-Based Selection

Models like XMoE, proposed by Yuanhang Yang, use a simple threshold on routing scores. If an expert's score doesn't meet the bar, it isn't activated. This allows the model to scale its computational footprint down by over 50% for simple sequences without losing accuracy.

Heterogeneous Architectures

Dynamic allocation isn't just about how many experts we pick, but also what kind of experts we use. Traditional MoEs use a homogeneous set of experts (all the same size). New directions suggest a "big.LITTLE" approach for neural networks.

  • Zero-Computation Experts: Peng Jin introduced MoE++, which includes "experts" that perform no computation or simple identity shortcuts. This allows the router to explicitly "skip" heavy processing for filler words or punctuation.
  • Slimmable Experts: In MoSE, Nurbek Tastan proposes experts with variable widths. The model can choose not only which expert to use but also how wide (and thus how expensive) that expert should be for a given token.
  • Stratified Experts: Haoran Xu suggests Stratified MoE (SMoE), which organizes experts into different layers or "strata" based on the complexity of the features they handle, ensuring that rare technical terms get more parameters than common subwords.

Shifting the Budget to the Sequence

Perhaps the most intuitive framing of this idea is moving the "computational budget" from the token level to the sequence level. In Route Experts by Sequence, not by Token, Tiansheng Wen argues that we should give a sentence a total "allowance" of expert activations.

If a sentence is "The cat sat on the mat," the model might spend almost all of its activations on "cat" and "mat," while effectively ignoring the function words. This sequence-level Top-K (SeqTopK) approach is parameter-free and allows the model to naturally navigate the heterogeneity of human language, where information density is rarely distributed uniformly.

By treating computation as a dynamic resource rather than a static constraint, these adaptive frameworks allow sparse models to become truly "elastic"—stretching their capacity for the hardest problems while remaining lean for the easiest ones.

Papers in this version

Showing 60 of 70 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_expert_allocation_in_sparse_tran_8fa78953,
  title = {Dynamic Expert Allocation in Sparse Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-expert-allocation-in-sparse-transformers/rdr_8fa789535e7e47b981ac441f1729395d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_expert_allocation_in_sparse_tran_8fa78953_v1,
  title = {Dynamic Expert Allocation in Sparse Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-expert-allocation-in-sparse-transformers/rdr_8fa789535e7e47b981ac441f1729395d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}