Differentiable Soft Routing for Mixture of Experts

60 concepts38 papers1 proposal

Traditional Mixture-of-Experts (MoE) models scale up parameter counts by routing inputs to a sparse subset of specialized sub-networks ("experts"). Typically, a router uses a discrete top-$k$ operation to select which expert handles which token. While this sparsity saves compute during inference, it introduces a fundamental mathematical headache: the discrete selection step is non-differentiable.

To train these models, researchers traditionally rely on discrete gradient approximations and complex auxiliary load-balancing losses to prevent "expert collapse"—a failure mode where the router prematurely converges to favoring a few experts, leaving the rest untrained. Furthermore, strict capacity limits mean tokens are frequently dropped when an expert overflows.

To bypass these limitations, a growing body of research replaces discrete selection with differentiable soft routing. Instead of making an all-or-nothing choice, soft routing treats expert assignment as a continuous, weighted interaction, smoothing the optimization landscape and allowing gradients to flow unimpeded.


The Two Paradigms of Soft Routing

Differentiable routing generally falls into two categories: blending the outputs of the experts, or blending the parameters of the experts themselves.

Discrete Routing (Top-1):
Input ──> [ Router ] ──> Choose Expert A ──> [ Expert A ] ──> Output

Soft Output Blending:
Input ──> [ Router ] ──> Weights [0.7, 0.2, 0.1]
      ├──> [ Expert A ] ──* 0.7 ──┐
      ├──> [ Expert B ] ──* 0.2 ──┼──> Sum ──> Output
      └──> [ Expert C ] ──* 0.1 ──┘

Soft Parameter Merging:
Input ──> [ Router ] ──> Weights [0.7, 0.2, 0.1] ──> Blended Parameters ──> [ Dynamic Expert ] ──> Output

1. Soft Output Blending

The most direct approach is to process inputs through multiple experts and combine their outputs using continuous weights. For instance, Timon Willi et al. demonstrate that soft routing provides a smoother optimization landscape for deep reinforcement learning (RL), where discrete routing often causes severe gradient instability and creates "dormant neurons." In value-based RL agents, Johan Obando-Ceron et al. show that swapping standard dense layers for Soft MoEs unlocks stable parameter scaling, bypassing the fragile load-balancing requirements of sparse architectures.

Instead of routing individual tokens to individual experts, some architectures invert the mapping entirely. In Soft MoE setups, input tokens are aggregated into a set of processing slots using learned dispatch weights. Every expert then processes a weighted linear combination (a convex combination) of all tokens. Tianlin Liu et al. show that this approach achieves superior accuracy-to-compute trade-offs in vision models because it completely eliminates sorting and ranking operations, making it highly optimized for modern hardware. Youngseog Chung et al. analyze the implicit bias of this mechanism, noting that allowing each expert to receive information from the entire sequence simultaneously resolves the non-differentiability of token-choice routing.

2. Soft Parameter Merging

Rather than passing data through multiple physical expert blocks and averaging their outputs, an alternative strategy is to average the experts themselves. This paradigm computes a weighted average of expert parameters based on the input context, creating a single, dynamically customized layer for each forward pass.

Mohammed Muqeeth et al. introduce this concept via SMEAR (Soft Merging of Experts with Adaptive Routing). They demonstrate that blending parameters naturally utilizes all available experts without auxiliary losses. Because every expert contributes fractionally to the merged weights, every expert receives a gradient update on every step, preventing the system from collapsing into a sparse subset.

This parameter-blending intuition is highly effective for parameter-efficient fine-tuning (PEFT) and model upcycling:

  • Instruction Tuning: Ted Zadouri et al. apply soft merging to low-rank adapters (MoLoRA), showing that activating a weighted average of all available adapters allows the model to leverage collective knowledge effectively within tight parameter constraints.
  • Language Model Pre-training: Zexuan Zhong et al. introduce Lory, which scales fully differentiable expert merging to autoregressive language model pre-training, achieving automatic domain-level expert specialization.
  • Upcycling Attention: Qizhen (Irene) Zhang et al. use soft routing to upcycle pre-trained dense models into MoEs by converting both feed-forward and attention layers into experts, smoothing the integration of specialized attention patterns.

Why It Works: The Intuition of Continuous Landscapes

The shift from discrete top-$k$ selection to soft routing works because it replaces a step-function optimization landscape with a continuous, differentiable one.

Continuous Gradient Flow

In discrete top-$k$ routing, an expert either receives a token or it doesn't. If a router slightly adjusts its internal weights but not enough to change the final top-$k$ ranking, the gradient with respect to the unselected experts is zero. This makes it incredibly difficult for the router to learn from mistakes.

Soft routing ensures that auxiliary choices still receive a gradient signal. For example, Bowen Dong et al. propose a smooth top-$k$ approximation called MaxScore, which includes a decaying term for non-selected elements. This ensures that secondary expert choices receive sufficient gradient signals, preventing the primary choice from mathematically dominating and starving the rest of the network of updates. Similarly, Ziteng Wang et al. replace discrete selection with a rectified linear unit (ReLU) activation function in ReMoE, creating a fully differentiable gate that allows gradients to flow continuously through the assignment process.

Elimination of Token Dropping

To run efficiently on hardware accelerators, standard sparse MoEs enforce a strict "expert capacity" limit. If a domain-specific expert receives more tokens than its capacity allows, the excess tokens are dropped and fail to be processed by their preferred block.

Soft routing mechanisms—such as the slot-based aggregation used by Umberto Cappellazzo et al. for audio transformers and Hang Zou et al. for multimodal face attack detection—bypass this entirely. Because inputs are mapped to processing slots via continuous dispatch weights, every token contributes to the expert inputs, maintaining a fully differentiable pipeline without the risk of data loss.

Robustness to Non-Stationarity

In environments where data distributions shift rapidly, discrete routing changes can cause catastrophic training failures. In reinforcement learning, Di Zhang et al. identify "Router Shift"—where expert routing changes too drastically between policy updates—as a primary cause of sudden reward collapse. They mitigate this by using a soft trust-weighting mechanism that allows the model to refine expert selection smoothly.

For continuous-time modeling, Xu Zhang et al. show that dense, soft routing is essential for maintaining stability when generating continuous time-series data from irregular observations, as discrete jumps in expert selection break the underlying differential equations of the system.


Applications Across Domains

By providing a stable, expressive way to share parameters, differentiable soft routing has solved domain-specific challenges across machine learning:

  • Multi-Task & Zero-Shot Learning: Poorya ZareMoodi et al. use soft-sharing via a routing sub-network to optimize the flow of information across linguistic tasks in low-resource machine translation, preventing task interference. For large-scale classification, Yuchen Guo demonstrates that a soft mixture of transformations breaks the rank barriers of traditional bilinear embeddings, significantly improving zero-shot learning across thousands of classes.
  • Multi-Agent Coordination: Yang Yu et al. employ soft routing in cooperative multi-agent reinforcement learning, allowing agents to share a pool of specialized modules while maintaining behavioral diversity.
  • Computer Vision & Adaptation: In medical image segmentation, Weili Jiang et al. use a soft routing strategy with fractional weights to combine residual experts and repair topological errors in complex structures. For continuous test-time adaptation in changing environments (e.g., shifting weather in stereo depth estimation), Jueun Ko et al. employ a sigmoid-based gating function to softly blend experts, preserving representational capacity without forgetting.
  • Hierarchical Modeling: Szymon Plotka et al. combine soft routing with state-space models (Mamba) to explicitly capture local-to-global anatomical hierarchies in 3D medical images, ensuring smooth gradient flow across spatial scales.

By trading the strict compute savings of hard sparsity for the mathematical cleanliness of continuous functions, differentiable soft routing provides a robust framework for training modular, scalable, and stable neural networks.

Key Foundational Work

Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation (2018) — contributed 1 concept elements to this direction.
Multimodal Learning and Reasoning for Visual Question Answering (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_differentiable_expert_routing_a40d707d,
  title = {differentiable expert routing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-expert-routing/rdr_a40d707d76bc45bfa0b1f24acec0abfc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_differentiable_expert_routing_a40d707d_v1,
  title = {differentiable expert routing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-expert-routing/rdr_a40d707d76bc45bfa0b1f24acec0abfc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake