Dynamic Expert Allocation in Mixture of Experts

45 concepts38 papers1 proposal

Sparse Mixture-of-Experts (MoE) models have become a cornerstone of modern large-scale neural networks. By routing inputs to a subset of specialized "experts" rather than activating the entire network, MoEs scale parameter count without a proportional increase in computational cost.

However, traditional MoE architectures rely on a rigid constraint: fixed Top-$K$ routing. Whether a model is processing a highly complex logical clause or a simple punctuation mark, the router is forced to select exactly $K$ experts (typically $K=1$ or $K=2$) for every single token. This "one-size-fits-all" approach introduces a fundamental inefficiency, wasting valuable floating-point operations (FLOPs) on trivial inputs while starving ambiguous or information-dense tokens of the capacity they require.

To resolve this, a powerful paradigm shift has emerged: dynamic expert allocation. Instead of treating compute as a fixed per-token budget, these techniques allow the network to adaptively vary the number, size, or depth of activated experts based on the real-time complexity of the input data.


The Core Intuition: Matching Compute to Information Density

To understand why dynamic allocation works, consider how information is distributed across natural language or images. In a typical sentence, tokens like the, of, or . carry minimal semantic uncertainty. Conversely, rare nouns, technical jargon, or words that trigger complex logical transitions require significant contextual processing.

When a model is forced to use a static number of experts, two points of failure occur:

  1. Over-computation: Simple tokens consume full expert capacity, leading to unnecessary memory bandwidth usage and latency.
  2. Under-computation: Complex tokens are bottlenecked by the strict parameter cap, limiting the model's representational capacity on difficult tasks.

Dynamic expert allocation breaks this bottleneck. By allowing the computational path to expand or contract on a token-by-token basis, the model naturally aligns its hardware expenditure with the inherent difficulty of the task.


Mechanisms for Variable-Capacity Routing

Researchers have developed several distinct algorithmic strategies to transition from static Top-$K$ selection to variable, token-adaptive routing.

1. Threshold-Based Activation

Instead of sorting routing scores and picking the top $K$ values, routers can evaluate scores against a dynamic or static threshold.

2. Learnable Allocators and Auxiliary Networks

Rather than relying on hardcoded heuristics or thresholds, a model can learn to predict token difficulty directly.

3. Zero-Cost Sinks and Null Experts

An elegant way to achieve dynamic compute allocation without modifying the underlying hardware-optimized Top-$K$ sorting operations is to introduce "dummy" experts into the routing pool.

4. Differentiable ReLU Gating

Traditional Top-$K$ selection relies on a non-differentiable argmax or sorting operation, making routing decisions notoriously difficult to optimize.

5. Inverting the Paradigm: Expert Choice

Instead of letting tokens choose which experts they want to visit, the routing mechanism can be inverted so that experts choose the tokens they are best equipped to process.

  • Heterogeneous Token Processing: Yanqi Zhou et al. pioneer this in Mixture-of-Experts with Expert Choice Routing. Because each expert independently selects its top tokens from a batch, high-value or highly complex tokens are naturally selected by multiple experts simultaneously, while simpler tokens receive fewer computational resources.

Why It Works: Evidence Across Domains

The intuition behind dynamic expert allocation is validated across a wide variety of problem domains, proving that variable compute allocation consistently outperforms uniform routing.

Linguistic Complexity and Efficiency

In natural language processing, forcing uniform compute wastes training time. Jiamin Li 0002 et al. demonstrate in Adaptive Gating in Mixture-of-Experts based Language Models that replacing fixed-$K$ routing with a flexible activation scheme reduces training time by up to 22.5% because the model allocates multiple parameters only to linguistically ambiguous or information-dense tokens.

Mitigating Long-Tailed Distributions

Real-world datasets are heavily imbalanced, featuring a few common "head" classes and a vast majority of rare "tail" classes. Standard models often get overwhelmed by head classes.

In Long-tailed Recognition by Routing Diverse Distribution-Aware Experts, XuDong Wang et al. show that dynamic routing elegantly solves this problem. Common samples are processed efficiently by a single expert, whereas rare or ambiguous samples trigger additional experts to act as a committee, reducing variance and boosting accuracy on the tail.

Managing Data Heterogeneity

In decentralized settings like Federated Learning, different clients possess vastly different data distributions. Lei Wang et al. show in Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning that allowing the model to dynamically adjust the number of active experts per input significantly improves performance across heterogeneous client silos by scaling specialized knowledge on demand.


Heterogeneous and Elastic Architectures

Dynamic allocation is not restricted to changing the number of active experts; it can also adapt the size, depth, or sub-network architecture allocated to an input.

Varying Expert Sizes

Instead of uniform expert dimensions, layers can house a mixture of small and large experts. Ganesh Jawahar proposes AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation, which routes complex tokens to larger experts and simpler tokens to smaller ones, drastically reducing total FLOPs while maintaining translation quality. Similarly, Haoran Xu et al. organize experts into stratified layers in Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity, ensuring high-frequency tokens consume fewer parameters overall.

Hardware-Aware Elasticity

In production environments, hardware constraints change dynamically based on concurrent workloads. Ruisi Cai et al. introduce Flextron: Many-in-One Flexible Large Language Model, a framework that evaluates token complexity to determine the optimal sub-network size for processing. This allows a single deployed model to dynamically scale its active parameters down during high-concurrency spikes or up when full capacity is required.

For vision tasks, Xingkui Zhu develops MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks, which upcycles dense models into adaptive MoEs by directing high-impact tokens to specialized experts and background noise to smaller, universal experts.


Extending Dynamics Beyond Tokens

The philosophy of dynamic allocation extends beyond individual tokens to encompass test-time scaling and global spatio-temporal competition.

Test-Time Scaling

Instead of fixing the model's computational footprint during training, the expert count can be treated as a searchable dimension during inference. Fan Ma et al. demonstrate in Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time that varying the number of activated experts at test time allows the model to explore diverse, complementary reasoning paths, unlocking better solutions for complex reasoning tasks without retraining.

Global Competition in Diffusion

In generative models like Diffusion Transformers, computational difficulty varies across both spatial regions (e.g., detailed subjects vs. empty backgrounds) and denoising timesteps (e.g., early structural formulation vs. late fine-detail polishing).

[Input Batch: Tokens across Space & Time]
                   │
                   ▼
     ┌───────────────────────────┐
     │ Global Competition Router │
     └─────────────┬─────────────┘
                   │
         ┌─────────┴─────────┐
         ▼                   ▼
┌─────────────────┐ ┌─────────────────┐
│ Complex Tokens  │ │ Simple Tokens   │
│ (High Detail /  │ │ (Background /   │
│  Early Steps)   │ │  Late Steps)    │
└────────┬────────┘ └────────┬────────┘
         │                   │
         ▼                   ▼
┌─────────────────┐ ┌─────────────────┐
│ Activate        │ │ Activate        │
│ Multiple Experts│ │ Fewer Experts   │
└─────────────────┘ └─────────────────┘

To address this, Yike Yuan et al. propose Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts. By allowing all tokens to compete globally across spatial regions and denoising timesteps for a shared pool of expert resources, the model concentrates its computational power precisely where the generation task is hardest, optimizing the overall quality-to-compute ratio.

Key Foundational Work

How Many Layers and Why? An Analysis of the Model Depth in Transformers (2021) — contributed 2 concept elements to this direction.
Dense-to-Sparse Gate for Mixture-of-Experts (2021) — contributed 1 concept elements to this direction.
Universal Transformers (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_adaptive_expert_selection_80a63e3b,
  title = {adaptive expert selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-expert-selection/rdr_80a63e3bbc2649efb4a8a4061bf14bb6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_adaptive_expert_selection_80a63e3b_v1,
  title = {adaptive expert selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-expert-selection/rdr_80a63e3bbc2649efb4a8a4061bf14bb6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake