Expert-Choice Routing for Sparse Mixture of Experts

This direction is merged into another direction: expert collapse, expert-choice routing. Its last version stays readable and citable here.

Version 1 of 148 papers59 conceptsfrozen 2026-10-04

As Large Language Models (LLMs) scale toward trillions of parameters, the computational cost of training and inference becomes a "memory wall." Sparse Mixture-of-Experts (MoE) architectures, as reviewed by Jeff Dean, Barret Zoph, and William Fedus in A Review of Sparse Expert Models in Deep Learning, offer a way out: they decouple a model's total parameter count from the computation required per token by activating only a small subset of "experts" for any given input.

However, the efficiency of these models hinges entirely on routing—the mechanism that decides which tokens go to which experts. Traditional routing often fails because it treats tokens as the primary decision-makers. A new direction in the field flips this perspective, letting the experts choose the tokens instead.

The Failure of Token-Choice Routing

In traditional "Token-Choice" routing, each token independently selects its top-$k$ preferred experts based on a router's score. While intuitive, this approach suffers from two major flaws:

  1. Load Imbalance: Tokens are "greedy." They often cluster around a few popular experts, leaving others idle. This creates hardware bottlenecks where some GPUs are overloaded while others wait.
  2. The Auxiliary Loss Trap: To fix imbalance, researchers typically add "load-balancing losses" to force tokens toward underutilized experts. These losses are finicky to tune and often conflict with the primary objective of model accuracy.

If an expert's capacity is exceeded, tokens are simply dropped, leading to information loss. If an expert is under-capacity, the hardware performs "padding," wasting FLOPs on zeros.

Flipping the Script: Expert-Choice Routing

To solve these issues, researchers like Yanqi Zhou and Nan Du introduced Expert-Choice (EC) routing in Brainformers: Trading Simplicity for Efficiency.

In EC routing, the experts are the active agents. Each expert is assigned a fixed capacity (e.g., $k$ tokens per batch) and selects the $k$ tokens from the entire batch for which it has the highest affinity scores.

This inversion provides three immediate benefits:

  • Guaranteed Load Balancing: Because every expert picks exactly $k$ tokens, the workload across all hardware accelerators is perfectly uniform by design. No auxiliary losses are required.
  • Variable Token Depth: In token-choice, every token gets exactly $k$ experts. In expert-choice, a "difficult" or highly informative token might be selected by many experts, while a redundant background token (like a period or a blank image patch) might be skipped entirely.
  • Efficiency: As demonstrated in EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing by Tao Lei, this mechanism allows models to scale to nearly 100 billion parameters with minimal inference overhead.

Routing as a Global Optimization Problem

While Expert-Choice is efficient, it can be seen as a "greedy" version of a larger mathematical problem. Mike Lewis and Luke Zettlemoyer proposed the BASE (Balanced Assignment of Sparse Experts) layer in BASE Layers: Simplifying Training of Large, Sparse Models.

Instead of simple top-$k$ selection, they frame routing as a linear assignment problem. By using the auction algorithm, the model finds a global mapping that maximizes the total affinity scores across the entire batch while ensuring every expert processes an equal number of tokens. This algorithmic guarantee of balance eliminates the need for hyperparameters associated with penalty terms.

More recently, Bowen Dong extended this in Maximum Score Routing For Mixture-of-Experts by formulating routing as a minimum-cost maximum-flow problem. This approach ensures that multi-expert routing actually utilizes diverse experts rather than redundantly assigning a token to the same expert multiple times.

Dynamic Compute and Mixture-of-Depths

The logic of expert-choice routing has paved the way for even more radical efficiency gains. In Mixture-of-Depths, David Raposo and Timothy Lillicrap apply this "capacity-first" thinking to the entire layer.

Instead of routing between experts within a layer, the model routes tokens around the layer itself. By setting a static compute budget (e.g., only 50% of tokens can be processed by a specific Transformer block), the router selects only the most "important" tokens for deep processing. This allows the model to dynamically allocate FLOPs where they are most needed, matching baseline performance while significantly increasing inference speed.

Synergy with Instruction Tuning

A surprising finding from Sheng Shen and Le Hou in Mixture-of-Experts Meets Instruction Tuning is that these sparse, expert-choice-driven models are uniquely suited for instruction tuning.

While MoE models sometimes struggle with narrow, task-specific finetuning, they benefit significantly more from broad instruction tuning than dense models. This synergy allowed a 32B parameter MoE model to outperform a 62B dense model using only one-third of the FLOPs, suggesting that the "expert" specialization enabled by balanced routing is exactly what is needed to master diverse, instruction-following tasks.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_expert_choice_routing_for_sparse_mixture_02c1679b,
  title = {Expert-Choice Routing for Sparse Mixture of Experts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-choice-routing-for-sparse-mixture-of-experts/rdr_02c1679b55bb4ce8af4733ad9b6f8de4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_expert_choice_routing_for_sparse_mixture_02c1679b_v1,
  title = {Expert-Choice Routing for Sparse Mixture of Experts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-choice-routing-for-sparse-mixture-of-experts/rdr_02c1679b55bb4ce8af4733ad9b6f8de4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}