Continuous Routing for Stable Mixture of Experts

This direction is split into other directions: differentiable expert routing. Its last version stays readable and citable here.

Version 1 of 1128 papers176 conceptsfrozen 2026-10-04

Traditional Mixture of Experts (MoE) models operate like a high-stakes sorting facility. When a token arrives, a "router" makes a discrete choice: it sends the token to Expert A or Expert B. This "Top-k" routing is the engine behind massive models like Mixtral, allowing them to scale parameters without a linear increase in compute.

However, this discrete selection introduces a fundamental mathematical friction. Because the choice to pick an expert is a "hard" operation (like a step function), it is non-differentiable. This forces researchers to rely on complex workarounds—like auxiliary load-balancing losses—to prevent "expert collapse," where the model ignores most experts and overloads a few. Furthermore, if an expert's "bin" is full, tokens are simply dropped, leading to information loss.

A new direction in research, exemplified by Soft MoE and Mixture of Tokens, proposes a shift from discrete "picking" to continuous "blending." By making the routing process fully differentiable, these models achieve greater stability, better scaling, and eliminate the need for heuristic load balancing.

The Intuition of Soft Assignment

In a standard sparse MoE, tokens are routed to experts. In a "Soft" or "Continuous" MoE, the perspective is flipped: experts pull from the tokens.

As Joan Puigcerver and colleagues describe in From Sparse to Soft Mixtures of Experts, the model defines a fixed number of "slots" for each expert. Instead of assigning a single token to a slot, each slot becomes a weighted average of all tokens in the input batch.

Imagine a classroom where, instead of one student being called to the board, the teacher asks for a "representative" answer that is 40% of what Student A knows, 30% of Student B, and so on. Because every token contributes to every slot (even if the weight is near zero), the entire path from input to output is differentiable. This allows the router to be trained using standard backpropagation, just like any other layer in a Transformer.

Differentiable Selection Mechanisms

While "Soft MoE" uses weighted averages of tokens, other researchers focus on making the expert selection itself smooth.

  • DSelect-k: Hussein Hazimeh introduced DSelect-k, which replaces the non-smooth Top-k operation with a continuously differentiable "sparse gate." This allows the model to learn which experts to select while maintaining a smooth gradient flow, which is particularly effective in multi-task learning and recommendation systems.
  • ReLU and Entmax Routing: In ReMoE, Ziteng Wang uses ReLU-based gates to create a differentiable path that still allows for sparsity. Similarly, James Oldfield utilizes sparse, differentiable functions like entmax in Multilinear Mixture of Experts to achieve specialization without the "jitter" of discrete switches.
  • MaxScore Routing: Bowen Dong proposes Maximum Score Routing, which uses a "SoftTopk" operator. This ensures that even experts who aren't the primary choice still receive a small gradient signal, preventing them from "starving" during training.

Why Continuous Routing Works

The move toward continuity solves three major problems that plague sparse models:

1. Training Stability

In sparse models, the router's decisions can change abruptly during training (router jitter), causing the experts to receive wildly different gradients from one step to the next. Continuous routing, as noted by Sebastian Jaszczur in Mixture of Tokens, provides a smoother loss landscape. This stability is especially crucial in Reinforcement Learning, where Ghada Sokar found that Soft MoEs unlock parameter scaling that was previously impossible due to optimization instabilities.

2. Eliminating Token Dropping

Sparse MoEs often have a "capacity limit" for experts to ensure hardware efficiency. If too many tokens are routed to one expert, the excess tokens are dropped. In a soft routing system, every token is integrated into the expert slots via weighted sums. As highlighted in Omni-SMoLA by Jialin Wu, this ensures that no information is lost and every expert is utilized in every forward pass.

3. Natural Load Balancing

Because the routing is differentiable, the model naturally learns to distribute work to minimize the loss. In Steel-LLM, Qingshui Gu demonstrates that this "convex combination" of tokens avoids the expert imbalance common in sparse methods, removing the need for the "hacky" auxiliary losses that researchers typically use to force experts to be active.

Expanding Beyond Feed-Forward Layers

The success of continuous routing has led to its application in more than just the standard MLP layers of a Transformer.

In BAM!, Qizhen Zhang shows that "upcycling" dense models into MoEs is more effective when using soft routing for attention layers. Similarly, Soft Mixture of Adapters (Soft-MoA) by Umberto Cappellazzo applies this logic to fine-tuning, where weighted combinations of adapter "experts" allow for more nuanced feature fusion than picking a single adapter.

By treating expert selection as a continuous optimization problem rather than a discrete sorting task, this research direction provides a path toward models that are not only larger and more efficient but also significantly more robust and easier to train.

Papers in this version

Showing 60 of 128 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_routing_for_stable_mixture_of_60e37f25,
  title = {Continuous Routing for Stable Mixture of Experts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-routing-for-stable-mixture-of-experts/rdr_60e37f25fcef4aa6b26b6da0a73c0c72},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_continuous_routing_for_stable_mixture_of_60e37f25_v1,
  title = {Continuous Routing for Stable Mixture of Experts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-routing-for-stable-mixture-of-experts/rdr_60e37f25fcef4aa6b26b6da0a73c0c72/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}