Standard Mixture-of-Experts (MoE) architectures scale the parameter count of large language models without a proportional increase in computational cost. They achieve this by routing each input token to a subset of specialized sub-networks, known as "experts." However, conventional MoE models rely on a rigid fixed-$k$ routing mechanism: every single token activates exactly $k$ experts (typically $k=1$ or $k=2$), regardless of whether the token is a simple punctuation mark or a complex technical term.
This uniform compute allocation introduces a fundamental inefficiency. Forcing simple tokens to pass through multiple large experts wastes billions of floating-point operations (FLOPs), while restricting highly complex tokens to the same fixed budget limits the model's capacity where it is needed most.
To resolve this, a rapidly growing body of research focuses on dynamic expert routing. By treating the number of active experts as a flexible, token-dependent variable, these techniques match computational density directly to the informational complexity of the input.
The Core Mechanism: Breaking the Fixed-k Constraint
The primary challenge in dynamic routing is moving away from the standard TopK operator, which is non-differentiable and inherently rigid. Researchers have developed two main paradigms to achieve variable expert activation: threshold-based gating and learnable sparsity parameters.
Threshold-Based Gating
Instead of selecting a fixed number of experts, the router evaluates the routing scores (usually probabilities from a softmax layer) against a dynamic or static threshold. If only one expert dominates the probability distribution, only that expert is activated.
- Yang et al. implement an adaptive, threshold-based router in their XMoE framework. By replacing a few large experts with many fine-grained "small experts," tokens activate only the precise parameters they require, cutting FLOPs by over 50%.
- J. Li et al. introduce an Adaptive Gating strategy that dynamically decides whether a token needs one or two experts based on linguistic complexity, accelerating training by up to 22.5%.
Learnable and Auto-Tuned Sparsity
Rather than relying on manual heuristics or hardcoded thresholds, recent architectures formulate expert selection as a continuous, learnable optimization problem.
- Yue et al. propose Ada-K Routing, which uses a learnable allocator to dynamically scale $k$ per token, reducing computational overhead by 25% while improving accuracy.
- Guo et al. develop DynMoE, an auto-tuning approach that frames expert selection as a multi-label classification problem, automatically discovering both the optimal total number of experts and the per-token allocation across vision and language tasks.
- Zhuang et al. introduce LD-MoLE, a fully differentiable routing mechanism for Mixture of LoRA Experts. It utilizes a learnable sparsity parameter and a closed-form projection method to adaptively determine expert counts at every layer.
Bypassing Compute with Zero-Computation Experts
An elegant and hardware-friendly alternative to altering the routing algorithm itself is the introduction of null experts or sink experts. This approach preserves the standard, highly optimized fixed-$k$ routing infrastructure but populates the expert pool with modules that perform zero computation.
Standard Top-2 Routing:
Token ("the") ---> [ Router ] ---> Expert 3 (Compute) & Expert 7 (Compute) [High Waste]
Null-Expert Routing:
Token ("the") ---> [ Router ] ---> Expert 3 (Compute) & Null Expert (No-Op) [50% Compute Saved]
When a token encounters a semantically shallow context, the router naturally assigns its top choices to these dummy modules, effectively bypassing the heavy matrix multiplications of standard feed-forward networks (FFNs).
- Zeng et al. demonstrate this with AdaMoE, showing that adding zero-computation modules allows the router to naturally drop unnecessary real experts, saving up to 15% in FLOPs.
- Jin et al. expand on this concept with MoE++, demonstrating that zero-computation experts accelerate training and inference by absorbing common punctuation and high-frequency linguistic fragments.
- Yan et al. introduce TC-MoE, which augments the expert space with a ternary choice mechanism ($\{-1, 0, 1\}$). Tokens can route to a zero-cost "sink" expert or even dynamically invert an expert's output, boosting accuracy while cutting overhead.
Macro-Level Budgets: Sequences, Chunks, and Layers
While token-level adaptivity is highly granular, it introduces significant engineering challenges. Token-by-token routing causes irregular memory access patterns and load-balancing bottlenecks across GPUs. To mitigate this, several researchers propose shifting the dynamic allocation budget to a more macro level.
Sequence and Chunk-Level Routing
Instead of enforcing a strict budget per token, the computational budget can be managed across an entire sequence or a contiguous block of tokens.
- Wen et al. introduce SeqTopK, a parameter-free modification that shifts the expert budget from the token level to the sequence level. This allows the model to globally spend its expert budget on "hard" tokens within a prompt while starving "easy" ones, avoiding complex architectural changes.
- Song et al. design BlockFFN to optimize MoE deployment on edge devices. It enforces chunk-level activation sparsity, ensuring that consecutive groups of tokens reuse the same small set of parameters. This enables efficient memory prefetching and cache management.
Cross-Layer Heterogeneity
Dynamic routing behavior is not uniform across a model's depth. Wengang Li et al. analyze cross-layer routing decisions and reveal that MoE routing is an entangled process where early layers exert a long-range, dominant influence on deeper layers. This suggests that uniform expert capacities across all layers are inherently inefficient, paving the way for layer-wise capacity allocation.
Post-Training and Test-Time Adaptivity
A major advantage of dynamic expert allocation is that it can often be applied to existing, pre-trained models without requiring expensive retraining from scratch.
Post-Training Pruning and Quantization
- Zhong et al. introduce AdapMoE, an algorithm-system co-design that dynamically prunes active experts during inference based on a "sensitivity" metric, pairing this with intelligent prefetching to accelerate memory-constrained devices.
- Chen et al. develop Ban&Pick, a plug-and-play post-training strategy that reinforces high-impact "key experts" while dynamically banning redundant ones based on token sensitivity.
- Wei Huang introduces the Mixture Compressor (MC) framework, which pairs dynamic expert pruning with mixed-precision quantization to compress models like Mixtral 8x7b down to 2.5 bits per parameter, protecting critical tokens while allowing execution on consumer hardware.
Test-Time Scaling
Dynamic routing can also be leveraged as a knob to scale model reasoning at inference time. Ma et al. propose Dynamic Experts Search (DES), which treats the expert count as a searchable dimension at test time. By dynamically varying the number of active experts during complex reasoning steps, the model explores diverse internal routing paths and generates higher-quality outputs without any parameter updates.
Why It Works: Aligning Compute with Information Theory
The success of dynamic expert allocation stems from its alignment with the underlying nature of data. In both natural language and multi-modal contexts (such as the multi-modal person re-identification framework proposed by Sun et al.), information density is highly variable.
When a dense model is converted into a dynamic MoE—as explored by Szatkowski et al.—it exploits natural activation sparsity. In any dense network, a large percentage of neuron outputs are zero or negligible for a given input. Fixed-$k$ routing forces the model to compute these negligible values anyway to satisfy the structural constraints of the tensor operations.
Dynamic routing works because it provides a mathematical release valve. By allowing the routing matrix to collapse to a single expert (or zero) for predictable inputs, the model frees up its parameter capacity. The remaining FLOPs are preserved for high-entropy tokens, allowing the network to achieve the representational capacity of a massive model at a fraction of the operational cost.
Key Foundational Work
AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference (2024) — contributed 1 concept elements to this direction.
XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection (2024) — contributed 1 concept elements to this direction.
Harder Task Needs More Experts: Dynamic Routing in MoE Models (2024) — contributed 1 concept elements to this direction.