Dynamic Expert Routing in Mixture of Experts

24 concepts23 papers1 proposal

Standard Mixture-of-Experts (MoE) architectures scale the parameter count of large language models without a proportional increase in computational cost. They achieve this by routing each input token to a subset of specialized sub-networks, known as "experts." However, conventional MoE models rely on a rigid fixed-$k$ routing mechanism: every single token activates exactly $k$ experts (typically $k=1$ or $k=2$), regardless of whether the token is a simple punctuation mark or a complex technical term.

This uniform compute allocation introduces a fundamental inefficiency. Forcing simple tokens to pass through multiple large experts wastes billions of floating-point operations (FLOPs), while restricting highly complex tokens to the same fixed budget limits the model's capacity where it is needed most.

To resolve this, a rapidly growing body of research focuses on dynamic expert routing. By treating the number of active experts as a flexible, token-dependent variable, these techniques match computational density directly to the informational complexity of the input.


The Core Mechanism: Breaking the Fixed-k Constraint

The primary challenge in dynamic routing is moving away from the standard TopK operator, which is non-differentiable and inherently rigid. Researchers have developed two main paradigms to achieve variable expert activation: threshold-based gating and learnable sparsity parameters.

Threshold-Based Gating

Instead of selecting a fixed number of experts, the router evaluates the routing scores (usually probabilities from a softmax layer) against a dynamic or static threshold. If only one expert dominates the probability distribution, only that expert is activated.

  • Yang et al. implement an adaptive, threshold-based router in their XMoE framework. By replacing a few large experts with many fine-grained "small experts," tokens activate only the precise parameters they require, cutting FLOPs by over 50%.
  • J. Li et al. introduce an Adaptive Gating strategy that dynamically decides whether a token needs one or two experts based on linguistic complexity, accelerating training by up to 22.5%.

Learnable and Auto-Tuned Sparsity

Rather than relying on manual heuristics or hardcoded thresholds, recent architectures formulate expert selection as a continuous, learnable optimization problem.

  • Yue et al. propose Ada-K Routing, which uses a learnable allocator to dynamically scale $k$ per token, reducing computational overhead by 25% while improving accuracy.
  • Guo et al. develop DynMoE, an auto-tuning approach that frames expert selection as a multi-label classification problem, automatically discovering both the optimal total number of experts and the per-token allocation across vision and language tasks.
  • Zhuang et al. introduce LD-MoLE, a fully differentiable routing mechanism for Mixture of LoRA Experts. It utilizes a learnable sparsity parameter and a closed-form projection method to adaptively determine expert counts at every layer.

Bypassing Compute with Zero-Computation Experts

An elegant and hardware-friendly alternative to altering the routing algorithm itself is the introduction of null experts or sink experts. This approach preserves the standard, highly optimized fixed-$k$ routing infrastructure but populates the expert pool with modules that perform zero computation.

Standard Top-2 Routing:
Token ("the") ---> [ Router ] ---> Expert 3 (Compute) & Expert 7 (Compute)  [High Waste]

Null-Expert Routing:
Token ("the") ---> [ Router ] ---> Expert 3 (Compute) & Null Expert (No-Op)  [50% Compute Saved]

When a token encounters a semantically shallow context, the router naturally assigns its top choices to these dummy modules, effectively bypassing the heavy matrix multiplications of standard feed-forward networks (FFNs).

  • Zeng et al. demonstrate this with AdaMoE, showing that adding zero-computation modules allows the router to naturally drop unnecessary real experts, saving up to 15% in FLOPs.
  • Jin et al. expand on this concept with MoE++, demonstrating that zero-computation experts accelerate training and inference by absorbing common punctuation and high-frequency linguistic fragments.
  • Yan et al. introduce TC-MoE, which augments the expert space with a ternary choice mechanism ($\{-1, 0, 1\}$). Tokens can route to a zero-cost "sink" expert or even dynamically invert an expert's output, boosting accuracy while cutting overhead.

Macro-Level Budgets: Sequences, Chunks, and Layers

While token-level adaptivity is highly granular, it introduces significant engineering challenges. Token-by-token routing causes irregular memory access patterns and load-balancing bottlenecks across GPUs. To mitigate this, several researchers propose shifting the dynamic allocation budget to a more macro level.

Sequence and Chunk-Level Routing

Instead of enforcing a strict budget per token, the computational budget can be managed across an entire sequence or a contiguous block of tokens.

  • Wen et al. introduce SeqTopK, a parameter-free modification that shifts the expert budget from the token level to the sequence level. This allows the model to globally spend its expert budget on "hard" tokens within a prompt while starving "easy" ones, avoiding complex architectural changes.
  • Song et al. design BlockFFN to optimize MoE deployment on edge devices. It enforces chunk-level activation sparsity, ensuring that consecutive groups of tokens reuse the same small set of parameters. This enables efficient memory prefetching and cache management.

Cross-Layer Heterogeneity

Dynamic routing behavior is not uniform across a model's depth. Wengang Li et al. analyze cross-layer routing decisions and reveal that MoE routing is an entangled process where early layers exert a long-range, dominant influence on deeper layers. This suggests that uniform expert capacities across all layers are inherently inefficient, paving the way for layer-wise capacity allocation.


Post-Training and Test-Time Adaptivity

A major advantage of dynamic expert allocation is that it can often be applied to existing, pre-trained models without requiring expensive retraining from scratch.

Post-Training Pruning and Quantization

  • Zhong et al. introduce AdapMoE, an algorithm-system co-design that dynamically prunes active experts during inference based on a "sensitivity" metric, pairing this with intelligent prefetching to accelerate memory-constrained devices.
  • Chen et al. develop Ban&Pick, a plug-and-play post-training strategy that reinforces high-impact "key experts" while dynamically banning redundant ones based on token sensitivity.
  • Wei Huang introduces the Mixture Compressor (MC) framework, which pairs dynamic expert pruning with mixed-precision quantization to compress models like Mixtral 8x7b down to 2.5 bits per parameter, protecting critical tokens while allowing execution on consumer hardware.

Test-Time Scaling

Dynamic routing can also be leveraged as a knob to scale model reasoning at inference time. Ma et al. propose Dynamic Experts Search (DES), which treats the expert count as a searchable dimension at test time. By dynamically varying the number of active experts during complex reasoning steps, the model explores diverse internal routing paths and generates higher-quality outputs without any parameter updates.


Why It Works: Aligning Compute with Information Theory

The success of dynamic expert allocation stems from its alignment with the underlying nature of data. In both natural language and multi-modal contexts (such as the multi-modal person re-identification framework proposed by Sun et al.), information density is highly variable.

When a dense model is converted into a dynamic MoE—as explored by Szatkowski et al.—it exploits natural activation sparsity. In any dense network, a large percentage of neuron outputs are zero or negligible for a given input. Fixed-$k$ routing forces the model to compute these negligible values anyway to satisfy the structural constraints of the tensor operations.

Dynamic routing works because it provides a mathematical release valve. By allowing the routing matrix to collapse to a single expert (or zero) for predictable inputs, the model frees up its parameter capacity. The remaining FLOPs are preserved for high-entropy tokens, allowing the network to achieve the representational capacity of a massive model at a fraction of the operational cost.

Key Foundational Work

AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference (2024) — contributed 1 concept elements to this direction.
XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection (2024) — contributed 1 concept elements to this direction.
Harder Task Needs More Experts: Dynamic Routing in MoE Models (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_static_expert_allocation_fc5f1a5c,
  title = {static expert allocation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-expert-allocation/rdr_fc5f1a5c3be343b7b83e3981c1dec560},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_static_expert_allocation_fc5f1a5c_v1,
  title = {static expert allocation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-expert-allocation/rdr_fc5f1a5c3be343b7b83e3981c1dec560/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake