Decoupling Expert Routing with Independent Sigmoid Gating

This direction is merged into another direction: expert collapse. Its last version stays readable and citable here.

Version 1 of 125 papers40 conceptsfrozen 2026-10-04

Traditional Mixture-of-Experts (MoE) architectures rely on a "winner-take-all" philosophy. By using a softmax function to route tokens to experts, these models force a zero-sum competition: for one expert to gain importance, others must lose it. While this creates sparsity, it also introduces significant training instabilities, expert collapse, and a need for complex auxiliary load-balancing losses.

A growing body of research suggests a more stable alternative: replacing the competitive softmax with independent sigmoid gating. By treating expert selection as a set of individual binary decisions rather than a probability distribution, researchers are finding that MoE models become easier to train, more sample-efficient, and more collaborative.

The Zero-Sum Problem of Softmax Gating

In a standard MoE, the router uses a softmax function to ensure that the weights assigned to experts sum to one. As Róbert Csordás and colleagues argue in $\sigma$-MoE, this creates an "artificial competition." If the model wants to increase the contribution of Expert A, it is forced to decrease the contribution of Expert B, even if both are relevant to the input.

This coupling has three major downsides:

  1. Expert Collapse: Early in training, a single expert might receive slightly better gradients, causing the softmax to favor it. This creates a feedback loop where one expert "wins" all tokens, leaving others untrained.
  2. Optimization Complexity: Because the experts are statistically coupled, the gradient for one expert depends on the parameters of all others. This complicates the optimization landscape.
  3. Rigid Sparsity: Softmax-based routing typically uses a fixed "Top-K" selection, forcing the model to activate exactly $K$ experts regardless of whether the input is simple or complex.

Decoupling Experts for Stability and Efficiency

By switching to sigmoid gating, each expert is assigned a score independently. In SwitchHead, Róbert Csordás demonstrates that this non-competitive selection eliminates the need for auxiliary load-balancing losses—the "hacks" usually required to keep MoE models from collapsing.

The benefits are not just practical but theoretical. Huy Nguyen, Nhat Ho, and Alessandro Rinaldo provided a rigorous proof in Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts. They showed that by removing the sum-to-one constraint, sigmoid gating allows the model to reach parametric convergence rates ($n^{-1/2}$) that softmax gating often struggles to achieve. Essentially, the model learns the "true" expert parameters much faster because the experts aren't constantly fighting for probability mass.

This decoupling is also central to the success of DeepSeekMoE. By using normalized sigmoid gating, the model reduces the statistical coupling between experts, allowing it to scale to a massive number of specialized experts without the training volatility seen in earlier sparse models.

From Competition to Collaboration

When experts are no longer forced to compete, they begin to function more like independent neurons in a dense layer. This shift enables several new capabilities:

  • Adaptive Activation: In the LTE (Learn-To-be-Efficient) framework, Haizhong Zheng and co-authors use sigmoid routing to let the model decide how many experts to activate. A complex token might trigger four experts, while a simple one might trigger only one (or zero, if the base weights suffice).
  • Expert Cooperation: Leiyu Pan and the team behind Finedeep found that sigmoid routing fosters "cooperative contributions." Instead of a winner-take-all dynamic, multiple experts can work together to represent a single token, increasing the model's overall activation density and representation capacity.
  • Handling Heterogeneity: Sigmoid gating is particularly effective when experts have different architectures. In MoA (Mixture-of-Adapters), Jie Cao uses sigmoid routing to integrate diverse modules like LoRA and Prompt Tuning, allowing the model to leverage the unique strengths of each without forcing an unnatural competition between them.

Scaling to Complex Scenes

The move toward independent gating is also unlocking new frontiers in 3D and multi-modal generation. Yunhan Yang and colleagues introduced MoCA, which uses sigmoid-based routing to manage up to 32 distinct components in a 3D scene. In this high-component regime, a softmax would suffer from vanishing gradients as the probability mass is spread too thin. Sigmoid gating maintains independent scores for every component, ensuring that even in complex scenes, the router can make sharp, stable decisions about which parts of the model to activate.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoupling_expert_routing_with_independe_162fdfbe,
  title = {Decoupling Expert Routing with Independent Sigmoid Gating},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-expert-routing-with-independent-sigmoid-gating/rdr_162fdfbeb13346b2a6fe1652c314afc9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoupling_expert_routing_with_independe_162fdfbe_v1,
  title = {Decoupling Expert Routing with Independent Sigmoid Gating},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-expert-routing-with-independent-sigmoid-gating/rdr_162fdfbeb13346b2a6fe1652c314afc9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}