Sparsely activated Mixture-of-Experts (MoE) architectures allow neural networks to scale their parameter count exponentially without a corresponding increase in computational cost. By routing each input token to only a small subset of specialized sub-networks ("experts"), MoEs keep the per-token floating-point operations (FLOPs) constant while dramatically expanding model capacity.
However, the traditional mechanism for distributing tokens across these experts suffers from a fundamental coordination problem. A rapidly growing body of research addresses this bottleneck by inverting the routing logic entirely: shifting agency from the tokens to the experts.
The Bottleneck of Token-Centric Routing
In standard MoE implementations, routing operates via a Token Choice mechanism. Each token independently evaluates its affinity for every expert and selects the top-$k$ sub-networks to process its representation.
Because tokens make these decisions in isolation, this approach frequently triggers severe load imbalances. Natural language and image data exhibit power-law distributions; certain tokens (such as punctuation marks or common visual backgrounds) recur frequently and map to the same preferred experts. Consequently, a few "popular" experts become heavily congested, while others remain starved for data.
To prevent hardware from idling while waiting for overloaded experts, implementations typically enforce a strict expert capacity—a maximum buffer size for the number of tokens an expert can accept in a single batch. If the number of tokens selecting an expert exceeds this capacity, the excess tokens are simply dropped and bypassed to the next layer without expert processing.
To mitigate this, traditional models rely on auxiliary load-balancing losses added to the training objective. However, tuning these auxiliary losses is notoriously difficult. If the loss weight is too low, hardware utilization drops and tokens are lost; if it is too high, it overrides the model's primary objective, forcing unnatural token assignments and degrading overall predictive accuracy.
Inverting Agency with Expert Choice
To resolve this dilemma, Yanqi Zhou et al. proposed a fundamental inversion of the routing paradigm in Mixture-of-Experts with Expert Choice Routing. Instead of letting tokens choose experts, Expert Choice routing allows each expert to independently select the most relevant tokens from the entire input batch.
Traditional Token Choice:
Token 1 ──> [ Router ] ──> Chooses Expert A & Expert B (Uncoordinated)
Token 2 ──> [ Router ] ──> Chooses Expert A (Congestion!)
Expert Choice Inversion:
Expert A ──> [ Router ] ──> Picks Top-k Tokens from Batch (Guaranteed Load)
Expert B ──> [ Router ] ──> Picks Top-k Tokens from Batch (Guaranteed Load)
Mechanically, the router computes an affinity score matrix between all tokens in a batch and all experts. Instead of running a top-k operation across the expert dimension for each token, the system runs a global top-k operation across the token dimension for each expert. Each expert selects a fixed number of tokens, $c$, where $c$ is determined by the total batch size divided by the number of experts, multiplied by a capacity factor.
This simple inversion yields two profound advantages:
- Perfect Load Balancing by Design: Because every expert is allocated a fixed budget of exactly $c$ tokens, the computational workload is distributed perfectly across all hardware accelerators. The need for auxiliary load-balancing losses is completely eliminated, simplifying the optimization landscape.
- Variable Processing Depth: In token-choice routing, every token receives exactly $k$ experts. In expert-choice routing, because experts select tokens independently, highly informative or ambiguous tokens can be chosen by multiple experts simultaneously. Conversely, low-information tokens can be skipped entirely.
As demonstrated by Tianlin Liu et al. in Routers in Vision Mixture of Experts: An Empirical Study, this flexibility allows the network to allocate its computational budget adaptively, significantly outperforming token-choice baselines in vision tasks.
Generalizing Expert Choice Across Domains
The intuition of expert-centric token selection has proven highly versatile, prompting researchers to adapt the mechanism across various layers and training paradigms.
Upcycling and Architecture Search
Training sparse models from scratch can be computationally prohibitive. Aran Komatsuzaki et al. demonstrated in Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints that a pre-trained dense network can be efficiently converted into an MoE by copying its dense layers to initialize multiple experts. Integrating expert-choice routing during this "upcycling" phase ensures that the newly expanded parameter space is utilized uniformly from the start of continued training.
Taking structural flexibility further, Yanqi Zhou et al. developed Brainformers: Trading Simplicity for Efficiency, using evolutionary search to discover non-uniform blocks that interleave dense and sparse layers. By leveraging expert-choice gating within these discovered blocks, the authors achieved a $2\times$ acceleration in training convergence compared to standard sparse architectures.
Sparse Attention Mechanisms
The quadratic scaling of self-attention remains a primary bottleneck for processing long sequences. In Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing, Róbert Csordás et al. extended the expert-choice paradigm to the attention mechanism itself. By treating individual attention heads as experts, each head independently selects a fixed number of tokens to attend to from the sequence. This ensures a uniform computational load across all heads while allowing each head to specialize dynamically in distinct token relationships.
Diffusion and Sequence Models
The benefits of expert-choice routing extend beyond language modeling. In EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing, Haotian Sun et al. applied the mechanism to Diffusion Transformers. Instead of processing every image patch uniformly, experts select the most relevant patches across the generative steps, enabling efficient scaling up to 97 billion parameters.
Similarly, James Lee-Thorp and the authors of Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT combined expert-choice routing with fast linear mixing transformations, delivering substantial training and inference speedups for encoder-only architectures.
Alternative Paradigms for Load Balancing and Inference
While expert-choice routing solves load balancing algorithmically by reversing the selection direction, other researchers have tackled the underlying hardware and data bottlenecks through alternative routing strategies and structural optimizations.
Priority-Modulated Routing
When dealing with severe data imbalances—such as multimodal settings containing vastly more image tokens than text tokens—standard routing can inadvertently suppress minority data. To address this in Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts, Basil Mustafa et al. introduced priority-based routing. Instead of a first-come-first-served allocation, expert buffer capacity is granted to tokens with the highest routing weights, protecting minority modalities from being dropped.
A related concept is explored by Jiaming Ma et al. in Less but More: Linear Adaptive Graph Learning Empowering Spatiotemporal Forecasting, where a priority modulator is integrated into a sparse graph-expert system to prevent the model from collapsing into a few dominant experts.
Batch-Aware Inference Coordination
During the decoding phase of large language models, inference is heavily memory-bound. Fetching unique expert weights from high-bandwidth memory (HBM) to on-chip SRAM for a single token introduces severe latency overhead.
To mitigate this, Costin-Andrei Oncescu et al. developed Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining. This approach evaluates routing decisions collectively across an entire batch of requests. Tokens are allowed to "piggyback" on additional experts that have already been loaded into memory by other requests in the same batch, extracting extra model capacity without incurring additional memory transfer costs.
Similarly, the authors of XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference utilize batch-level expert selection to evaluate the aggregate utility of experts, minimizing the total footprint of unique weights that must be concurrently active.
Hardware-Level Sharding
Rather than altering the routing algorithm to fit the hardware, the load-balancing problem can also be solved by restructuring how the hardware stores the model. In Accelerating MoE Model Inference with Expert Sharding, Anne-Marie Kermarrec et al. introduced MoEShard. By splitting and sharding the weights of every expert across all available compute nodes, the system ensures that every processing unit executes an identical amount of work, neutralizing the performance penalties of skewed token routing entirely at the infrastructure level.
Key Foundational Work
Mixture-of-Experts with Expert Choice Routing (2022) — contributed 3 concept elements to this direction.
Brainformers: Trading Simplicity for Efficiency (2023) — contributed 2 concept elements to this direction.
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022) — contributed 1 concept elements to this direction.