A mixture-of-experts (MoE) layer contains many parallel sub-networks ("experts"). A small model, the router, sends each token to only a few of them, typically the top-$K$ by score. These layers often suffer from expert collapse: a few experts take almost all the tokens and the rest go silent. The usual fix is a balancing penalty that pushes the router toward uniform use. That penalty cannot tell an expert that is idle because it lost an early race from an expert that is idle because the data does not need it. Applied too strongly, it also forces experts to become copies of one another. Part of the difficulty is that the router learns almost nothing about the experts it skips. In the backward pass, top-$K$ selection passes zero gradient to unselected experts, and the gradient for selected ones is a local, jagged signal.
Interpretability research faced a similar problem with input gradients and solved it by averaging. SmoothGrad averages gradients over Gaussian-perturbed inputs, which is equivalent to differentiating a Gaussian-blurred version of the function. That blurring removes high-frequency gradient noise. We propose doing the same for the router: Gaussian-smooth the router's discrete top-$K$ choice and estimate the gradient of the smoothed loss from a few real counterfactual routings on a small token subsample. This estimator gives each skipped expert a usable signal: "would these tokens have been better off with you?" We use that signal in two ways. As a diagnostic, it separates starved-but-useful experts from legitimately unused ones. As a training signal, it should let the router revive starved experts without the homogenizing pressure of a strong balancing loss.
Core Claim
The quantity that separates pathological collapse from healthy specialization is the smoothed counterfactual utility of each expert on the tokens it does not receive. This quantity can be estimated cheaply enough to use during training. Write $z$ for a token's router scores, $S(z)$ for the set of $K$ experts chosen, and $\ell(S)$ for the token's loss under that routing. With $\varepsilon \sim \mathcal{N}(0,\sigma^2 I)$, Stein's identity gives the gradient of the Gaussian-blurred routing loss without differentiating through the selection:
$$ g_\sigma(z) \;=\; \nabla_z\, \mathbb{E}_\varepsilon\big[\ell(S(z+\varepsilon))\big] \;=\; \frac{1}{\sigma^2}\,\mathbb{E}_\varepsilon\Big[\big(\ell(S(z+\varepsilon)) - \ell(S(z))\big)\,\varepsilon\Big]. $$
Subtracting $\ell(S(z))$ reduces variance and does not change the expectation, because $\varepsilon$ has mean zero. We avoid re-running later layers when evaluating the loss under a perturbed routing. Instead we use the first-order change $\langle \partial\ell/\partial h,\, h(S') - h(S)\rangle$, where $h$ is the MoE layer's output and $\partial\ell/\partial h$ is already available in the ordinary backward pass. The extra cost is computing a few additional experts' outputs for a small fraction of tokens.
Averaging $-[g_\sigma]_e$ over tokens gives a per-expert utility $u_e$. A cold expert with $u_e>0$ would lower the loss if it were routed more tokens, so it is starved. A cold expert with $u_e \le 0$ is unused for a good reason.
Relation to Prior Work
Several papers motivate this proposal by locating the gap. Scaling Vision with Sparse Mixture of Experts describes collapse as a feedback loop that starts from small chance advantages at initialization. Advancing Expert Specialization for Better MoE and Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer show that strong balancing losses homogenize experts. Rethinking Convergence in MoE Training: The Role of Routing Sparsity bounds the optimization cost of discrete routing switches, which it calls "routing-induced discrepancy." It explicitly does not ask whether smoothing the routing choice damps those switches. Our estimator does exactly that, because it differentiates a function in which switches have been blurred.
The smoothing method itself comes from the interpretability literature. SmoothGrad: removing noise by adding noise introduced gradient averaging over noisy inputs. Smoothed Geometry for Robust Attribution shows the averaged gradient is the exact gradient of a Gaussian-convolved function. Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation supplies estimator and variance-reduction tools for non-differentiable functions. We also borrow a sanity-check methodology from Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values, which showed that gradient explanations can be insensitive to whether the weights are trained. We apply the same test to $u_e$: randomizing a cold expert's weights should change its utility, and a diagnostic that does not respond fails. AdaptGrad: Adaptive Sampling to Reduce Noise motivates setting $\sigma$ adaptively rather than by hand. Here we tie $\sigma$ to the gap between the $K$-th and $(K+1)$-th router scores, so smoothing reaches just past the selection boundary.
The closest MoE work splits into two groups.
The first group injects noise in the forward pass. "Stability by smoothing" in Towards Understanding the Mixture-of-Experts Layer in Deep Learning adds noise to the router to stabilize early exploration. Its analysis is limited to CNN experts on clustered data, with early stopping. S2MoE routes noise-augmented copies of token representations and adds a contrastive loss. Both use noise to change which experts are chosen. Neither uses noise to build a gradient estimator, so an unselected expert still receives zero learning signal. Statistical Advantages of Perturbing Cosine Router in Mixture of Experts adds deterministic constants to the router's normalization to fix an identifiability problem. That is a different mechanism, which we treat as a compatible router choice.
The second group improves the router's gradient. Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (Default MoE) gives the router a dense gradient by substituting a moving-average "default" output for each unselected expert. That substitute is the same for every token and lags behind training. Our estimator uses real, token-specific expert outputs, but only on a subsample. GRIN: GRadient-INformed MoE estimates gradients through sampled routing with a higher-order numerical scheme. It aims at an unbiased gradient of the sampled model, not at a per-expert starvation signal.
$\phi$-Balancing for Mixture-of-Experts Training replaces batch-level balancing with population-level congestion prices. It names scheduling routing noise by the balancing state as untested future work. Our proposal uses noise to measure utility instead of injecting it for exploration, and we test whether that measurement can scale the balancing price per expert.
Research Program
We will first build the estimator as a drop-in router-gradient module for a dropless top-$K$ MoE. On each step, a fraction $\rho$ of tokens (default $1/64$) also computes the next $m$ experts beyond the top-$K$ (default $m=4$). Perturbed routings that reach past these $K+m$ experts are truncated. With $n$ Gaussian samples per subsampled token (default 8), the first-order loss difference costs only inner products, because each perturbed routing is a recombination of already-computed expert outputs.
The router's gradient becomes the ordinary top-$K$ gradient plus $\lambda\, g_\sigma$ on subsampled tokens. We keep a $\phi$-balancing price as a hardware safeguard. In the main variant, each expert's price is scaled down when its utility $u_e$ (tracked as a moving average) indicates healthy specialization rather than starvation.
The analysis has three parts. First, we test whether $u_e$ predicts the true counterfactual: the loss change from forcibly routing a held-out batch to a cold expert. That ground truth is expensive to compute, but it is only needed for evaluation. Second, we run the randomization sanity check on $u_e$. Third, we measure whether routing-induced discrepancy, defined as the step-to-step change in loss from assignment flips following Zhu et al., falls under the smoothed estimator.
Baselines
The naive reference is a standard top-$K$ MoE with a Switch-style batch balancing loss, with its coefficient tuned. It shows how much collapse and homogenization a careful practitioner gets by default.
The strongest competitor on router learning signal is Default MoE. It targets exactly the missing gradient to unselected experts, at a reported 0.18% throughput cost, and it reaches target validation perplexity "9% to 15% faster" than top-$K$ baselines on FineWeb (Dense Backpropagation Improves Training for Sparse Mixture-of-Experts).
The strongest competitor on the balance–specialization trade-off is $\phi$-balancing. On Moonlight-16B-A3B fine-tuning it reports Math500 rising from 64.8% to 66.2% and HumanEval from 73.2% to 75.9% ($\phi$-Balancing).
For forward-noise methods we include noisy routing in the style of Towards Understanding the Mixture-of-Experts Layer in Deep Learning and S2MoE. The file has no reported S2MoE numbers on enwik8, text8 or WikiText-103 (no reported result found on Lacuna for S2MoE on these). We therefore rerun it ourselves.
For character-level modeling we anchor to the perturbed cosine router, which reports enwik8 bits per character of 1.197 against 1.213 for the vanilla cosine router (Statistical Advantages of Perturbing Cosine Router). We use it both as a baseline and as a router to which our estimator is added.
The key ablation is "exact dense gradient on the same subsample." It evaluates the same extra experts on the same tokens and backpropagates the ordinary soft-gate derivative, with no Gaussian smoothing. This separates the effect of smoothing from the effect of having real counterfactual outputs.
All methods share data, token budget, architecture, and wall-clock-matched compute. Our extra expert FLOPs are counted against our budget, and baselines get the equivalent as extra training tokens. Every method receives the same hyperparameter sweep size, about 12 configurations, covering its own coefficients. Beating a baseline means reaching matched validation perplexity with fewer compute-matched tokens, and doing so without a worse load Gini coefficient or worse expert similarity, across three seeds.
Why not just…?
Why not just add noise to the router, as prior work already does? Forward noise changes which experts get selected. The gradient still flows only through selected experts, so a starved expert learns about its own usefulness only when noise happens to pick it. Our forward-noise baselines use the same $\sigma$ schedule as our estimator. If they match our results, the benefit came from exploration, not from the estimator, and we will report it that way.
Why not just use Default MoE's moving-average outputs? This is the most serious threat, because it is nearly free. Its substitute output is identical for every token. It therefore cannot express that an expert would help some tokens and hurt others, which is exactly what distinguishes specialization from starvation. The decisive test is the utility-prediction experiment. If Default MoE's implied router gradient correlates with the true counterfactual as well as $u_e$ does, our extra compute is unjustified.
Why not just compute the exact dense gradient on a token subsample? This ablation could make the smoothing unnecessary. The exact soft-gate derivative is local. Smoothing instead measures the effect of actually switching an expert in or out, which is the discrete jump Zhu et al. identify as harmful. If the ablation matches our method on both convergence and discrepancy, we will drop the smoothing and report the simpler counterfactual-subsample method as the finding.
Why not just tune $\phi$-balancing? Population-level prices remove batch bias, but they still price all imbalance equally. The per-expert scaled variant tests whether utility-aware prices improve on uniform pricing at equal tuning budget.
Proposed Experiments
We start with the synthetic clustered task of Towards Understanding the Mixture-of-Experts Layer in Deep Learning, where ground-truth cluster–expert structure is known. That paper reports 99.46% for a nonlinear MoE. We deliberately bias the router at initialization to starve chosen experts and check whether $u_e$ flags them while leaving genuinely redundant experts unflagged. Next we train Switch-style models on enwik8, text8 and WikiText-103, where we measure bits per character, iterations to target loss, and routing-induced discrepancy. We then pretrain roughly 1B-parameter MoEs with 64 experts and $K=2$ on FineWeb (resource) against Default MoE, with zero-shot MMLU, ARC and HellaSwag as secondary readouts. If those results are positive, we will fine-tune Moonlight-16B-A3B under the $\phi$-balancing protocol on Math500 and HumanEval.
We consider three checks decisive. First, $u_e$ must predict true counterfactual loss change better than Default MoE's implied gradient. Second, $u_e$ must pass the weight-randomization sanity check. Third, the smoothed estimator must beat the unsmoothed dense-subsample ablation in compute-matched perplexity while showing lower routing-induced discrepancy. If the third check fails, the paper becomes a diagnostic contribution, not a training method.
References
- Scaling Vision with Sparse Mixture of Experts
- Advancing Expert Specialization for Better MoE
- Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
- Rethinking Convergence in MoE Training: The Role of Routing Sparsity
- SmoothGrad: removing noise by adding noise
- Smoothed Geometry for Robust Attribution
- Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation
- Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values
- AdaptGrad: Adaptive Sampling to Reduce Noise
- Towards Understanding the Mixture-of-Experts Layer in Deep Learning
- S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning
- Statistical Advantages of Perturbing Cosine Router in Mixture of Experts
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
- GRIN: GRadient-INformed MoE
- $\phi$-Balancing for Mixture-of-Experts Training