Sparse Mixture-of-Experts (MoE) models have become a cornerstone of modern large-scale neural networks. By routing inputs to a subset of specialized "experts" rather than activating the entire network, MoEs scale parameter count without a proportional increase in computational cost.
However, traditional MoE architectures rely on a rigid constraint: fixed Top-$K$ routing. Whether a model is processing a highly complex logical clause or a simple punctuation mark, the router is forced to select exactly $K$ experts (typically $K=1$ or $K=2$) for every single token. This "one-size-fits-all" approach introduces a fundamental inefficiency, wasting valuable floating-point operations (FLOPs) on trivial inputs while starving ambiguous or information-dense tokens of the capacity they require.
To resolve this, a powerful paradigm shift has emerged: dynamic expert allocation. Instead of treating compute as a fixed per-token budget, these techniques allow the network to adaptively vary the number, size, or depth of activated experts based on the real-time complexity of the input data.
The Core Intuition: Matching Compute to Information Density
To understand why dynamic allocation works, consider how information is distributed across natural language or images. In a typical sentence, tokens like the, of, or . carry minimal semantic uncertainty. Conversely, rare nouns, technical jargon, or words that trigger complex logical transitions require significant contextual processing.
When a model is forced to use a static number of experts, two points of failure occur:
- Over-computation: Simple tokens consume full expert capacity, leading to unnecessary memory bandwidth usage and latency.
- Under-computation: Complex tokens are bottlenecked by the strict parameter cap, limiting the model's representational capacity on difficult tasks.
Dynamic expert allocation breaks this bottleneck. By allowing the computational path to expand or contract on a token-by-token basis, the model naturally aligns its hardware expenditure with the inherent difficulty of the task.
Mechanisms for Variable-Capacity Routing
Researchers have developed several distinct algorithmic strategies to transition from static Top-$K$ selection to variable, token-adaptive routing.
1. Threshold-Based Activation
Instead of sorting routing scores and picking the top $K$ values, routers can evaluate scores against a dynamic or static threshold.
- Weight Thresholds: Xiaonan Nie et al. introduce this concept in Dense-to-Sparse Gate for Mixture-of-Experts, demonstrating that threshold-based gating allows more efficient compute allocation across varying input complexities.
- Cumulative Probability Thresholds: In XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection, Yuanhang Yang et al. propose selecting the minimum number of experts required to meet a cumulative probability target. If a token's routing distribution is highly confident and concentrated on one expert, only that expert is activated. If the token is ambiguous, the pool expands dynamically to capture the uncertainty.
- Relative Contribution: Filip Szatkowski et al. exploit activation sparsity in Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion by applying a relative contribution threshold, ensuring that experts are only activated if they meaningfully contribute to the token's representation.
2. Learnable Allocators and Auxiliary Networks
Rather than relying on hardcoded heuristics or thresholds, a model can learn to predict token difficulty directly.
- Predicting Expert Count: Tongtian Yue et al. develop Ada-K Routing: Boosting the Efficiency of MoE-based LLMs, which replaces static routing with a lightweight, learnable allocator module that explicitly determines the optimal number of experts for each token.
- Difficulty-Driven MoEfication: Kumari Nishu et al. implement a post-training framework in From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs that trains a small auxiliary neural network to predict the required computational capacity for a token before routing it, preventing the model from defaulting to uniform expert sizes.
3. Zero-Cost Sinks and Null Experts
An elegant way to achieve dynamic compute allocation without modifying the underlying hardware-optimized Top-$K$ sorting operations is to introduce "dummy" experts into the routing pool.
- Null Experts: Zihao Zeng et al. introduce AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models. By populating the routing pool with zero-computation modules, a standard Top-$K$ router can choose to send a simple token to multiple "null" experts, effectively reducing the active parameter load on the fly.
- Ternary Sinks: The authors of TC-MoE: Augmenting Mixture of Experts with Ternary Expert Choice expand the expert space by allowing outputs to be multiplied by $\{-1, 0, 1\}$. By routing unnecessary computation to a parameter-free expert that always outputs zero, the architecture achieves a variable-load mechanism within a fixed routing framework.
4. Differentiable ReLU Gating
Traditional Top-$K$ selection relies on a non-differentiable argmax or sorting operation, making routing decisions notoriously difficult to optimize.
- Smooth Sparsity: Ziteng Wang et al. introduce ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing, which swaps standard gating for a smooth, fully differentiable ReLU-based gate. Because ReLU naturally outputs exact zeros for negative inputs, the model learns to dynamically vary the active expert count per token based on input complexity. Chenyang Song et al. leverage a similar differentiable approach in BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity to enforce chunk-level activation sparsity for edge devices.
5. Inverting the Paradigm: Expert Choice
Instead of letting tokens choose which experts they want to visit, the routing mechanism can be inverted so that experts choose the tokens they are best equipped to process.
- Heterogeneous Token Processing: Yanqi Zhou et al. pioneer this in Mixture-of-Experts with Expert Choice Routing. Because each expert independently selects its top tokens from a batch, high-value or highly complex tokens are naturally selected by multiple experts simultaneously, while simpler tokens receive fewer computational resources.
Why It Works: Evidence Across Domains
The intuition behind dynamic expert allocation is validated across a wide variety of problem domains, proving that variable compute allocation consistently outperforms uniform routing.
Linguistic Complexity and Efficiency
In natural language processing, forcing uniform compute wastes training time. Jiamin Li 0002 et al. demonstrate in Adaptive Gating in Mixture-of-Experts based Language Models that replacing fixed-$K$ routing with a flexible activation scheme reduces training time by up to 22.5% because the model allocates multiple parameters only to linguistically ambiguous or information-dense tokens.
Mitigating Long-Tailed Distributions
Real-world datasets are heavily imbalanced, featuring a few common "head" classes and a vast majority of rare "tail" classes. Standard models often get overwhelmed by head classes.
In Long-tailed Recognition by Routing Diverse Distribution-Aware Experts, XuDong Wang et al. show that dynamic routing elegantly solves this problem. Common samples are processed efficiently by a single expert, whereas rare or ambiguous samples trigger additional experts to act as a committee, reducing variance and boosting accuracy on the tail.
Managing Data Heterogeneity
In decentralized settings like Federated Learning, different clients possess vastly different data distributions. Lei Wang et al. show in Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning that allowing the model to dynamically adjust the number of active experts per input significantly improves performance across heterogeneous client silos by scaling specialized knowledge on demand.
Heterogeneous and Elastic Architectures
Dynamic allocation is not restricted to changing the number of active experts; it can also adapt the size, depth, or sub-network architecture allocated to an input.
Varying Expert Sizes
Instead of uniform expert dimensions, layers can house a mixture of small and large experts. Ganesh Jawahar proposes AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation, which routes complex tokens to larger experts and simpler tokens to smaller ones, drastically reducing total FLOPs while maintaining translation quality. Similarly, Haoran Xu et al. organize experts into stratified layers in Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity, ensuring high-frequency tokens consume fewer parameters overall.
Hardware-Aware Elasticity
In production environments, hardware constraints change dynamically based on concurrent workloads. Ruisi Cai et al. introduce Flextron: Many-in-One Flexible Large Language Model, a framework that evaluates token complexity to determine the optimal sub-network size for processing. This allows a single deployed model to dynamically scale its active parameters down during high-concurrency spikes or up when full capacity is required.
For vision tasks, Xingkui Zhu develops MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks, which upcycles dense models into adaptive MoEs by directing high-impact tokens to specialized experts and background noise to smaller, universal experts.
Extending Dynamics Beyond Tokens
The philosophy of dynamic allocation extends beyond individual tokens to encompass test-time scaling and global spatio-temporal competition.
Test-Time Scaling
Instead of fixing the model's computational footprint during training, the expert count can be treated as a searchable dimension during inference. Fan Ma et al. demonstrate in Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time that varying the number of activated experts at test time allows the model to explore diverse, complementary reasoning paths, unlocking better solutions for complex reasoning tasks without retraining.
Global Competition in Diffusion
In generative models like Diffusion Transformers, computational difficulty varies across both spatial regions (e.g., detailed subjects vs. empty backgrounds) and denoising timesteps (e.g., early structural formulation vs. late fine-detail polishing).
[Input Batch: Tokens across Space & Time]
│
▼
┌───────────────────────────┐
│ Global Competition Router │
└─────────────┬─────────────┘
│
┌─────────┴─────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Complex Tokens │ │ Simple Tokens │
│ (High Detail / │ │ (Background / │
│ Early Steps) │ │ Late Steps) │
└────────┬────────┘ └────────┬────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Activate │ │ Activate │
│ Multiple Experts│ │ Fewer Experts │
└─────────────────┘ └─────────────────┘
To address this, Yike Yuan et al. propose Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts. By allowing all tokens to compete globally across spatial regions and denoising timesteps for a shared pool of expert resources, the model concentrates its computational power precisely where the generation task is hardest, optimizing the overall quality-to-compute ratio.
Key Foundational Work
How Many Layers and Why? An Analysis of the Model Depth in Transformers (2021) — contributed 2 concept elements to this direction.
Dense-to-Sparse Gate for Mixture-of-Experts (2021) — contributed 1 concept elements to this direction.
Universal Transformers (2019) — contributed 1 concept elements to this direction.