A mixture-of-experts model splits each layer into many small sub-networks ("experts") and uses a small gating network, the router, to send each input or token to only one or two of them. Training these models has a well-known failure called expert collapse. Either a few experts take all the traffic while the others starve, or the losses added to force balanced traffic make every expert learn nearly the same function. Explanation methods face a separate, equally well-documented problem. Feature attributions such as SHAP can produce contradictory rankings of important features for the same input and the same model. This happens even when the model is 99% confident (Explanation Multiplicity in SHAP).
This note argues that in mixture-of-experts models the two problems are linked through one measurable quantity. An attribution method works by probing a model with perturbed copies of an input. If the input sits near a point where the router is undecided between two experts, those perturbed copies get sent to different experts, and the explanation inherits the router's indecision. The size of that inherited uncertainty depends on two things: how often the router is near a tie, and how differently the competing experts actually compute. Both are shaped by the anti-collapse method used in training. We propose to measure this routing-induced explanation uncertainty directly, separate it from ordinary sampling noise in the explainer, and report explanations as a probability-weighted mixture over routing outcomes rather than as a single vector.
Core Claim
Fix an input $x$ and let $c$ denote a routing configuration, meaning the set of experts chosen at the routing layers that matter for $x$. A route-pinned attribution $\phi(x \mid c)$ is the explanation computed with the router frozen at $c$, so that every perturbed sample the explainer evaluates passes through the same experts. Let $p(c \mid x)$ be the router's probability of choosing $c$. The route-aware explanation is the mixture of the $\phi(x \mid c)$ under $p(c \mid x)$. Its uncertainty for feature $j$ splits by the law of total variance:
$$ \mathrm{Var}[\phi_j(x)] = \mathbb{E}_{c \sim p(c\mid x)}\big[\mathrm{Var}(\phi_j(x \mid c))\big] + \mathrm{Var}_{c \sim p(c\mid x)}\big[\mathbb{E}\,\phi_j(x \mid c)\big]. $$
The first term is within-route uncertainty, such as explainer sampling error or surrogate fit error. The second term is between-route uncertainty, which is routing-induced.
We make three claims. First, in trained mixture-of-experts models a substantial share of the rank churn that Explanation Multiplicity in SHAP attributes to "explainer-induced stochasticity" is actually between-route variance, concentrated on inputs near routing ties. Second, the between-route term behaves non-monotonically across anti-collapse regimes. Batch-level balancing losses push routing toward uniform and create many ties, but they also homogenize the experts, so ties change little. Specialization losses such as those in Advancing Expert Specialization for Better MoE produce fewer ties, but each tie flips the explanation sharply. Third, reporting the route-pinned mixture gives better-calibrated explanation intervals than resampling the explainer at equal compute. It also exposes a failure that existing interval methods cannot see: an MoE whose "which expert handled this" explanation is vacuous because the competing experts are near-identical.
Relation to Prior Work
Two papers motivate the gap. Variational Routing shows that deterministic routers jitter under small input noise and fixes this with a Bayesian router that improves output calibration. It never asks what that jitter does to explanations. Explanation Multiplicity in SHAP separates model-seed noise from explainer noise using top-$k$ Jaccard distance and rank-biased overlap. It treats the model as a black box, so it cannot isolate an internal discrete switch as a cause.
Three papers sit closest to this proposal but answer different questions. Interpretable Mixture of Experts makes routing itself the explanation, using utilization and diversity losses. It reports assignments deterministically and never checks whether the chosen experts truly differ. Adaptive Conformal Prediction via MoE Gating Similarity uses gating vectors to tighten intervals on predictions, not on attributions, and it assumes a healthy router. Toward Calibrated MoE Under Distribution Shift shows that soft mixtures hide many expert configurations behind identical outputs. That result supports our choice to condition on configurations rather than on outputs. Our proposal differs from all three in the object it bounds: the attribution vector, decomposed by routing outcome.
Other papers supply methods we reuse. For within-route uncertainty we use BayLIME's posterior over surrogate weights (BayLIME) and conformal surrogate intervals (Approximating Score-based Explanation Techniques Using Conformal Regression). For the training regimes we use Switch-style batch balancing, population-level φ-Balancing, and orthogonality-plus-variance losses (Advancing Expert Specialization).
One paper constrains the design. The finding in Mixture of Experts with Soft Nearest Neighbor Loss, that disentangling features before routing over-regularizes simple data, warns us that "more specialized" is not uniformly better. That is why we predict a non-monotone relationship rather than ranking the regimes.
Research Program
The work begins with a controlled model zoo. We train small mixture-of-experts classifiers under four routing regimes, five seeds each:
- tabular MLPs on ACS Income, German Credit and Diabetes, the datasets used in the multiplicity study;
- convolutional MoEs on CIFAR-10 and CIFAR-100;
- a synthetic multi-regime regression with linear, quadratic and cubic pieces, following the MoE-CP setup.
The four regimes are no balancing (deliberate collapse), Switch-style batch balancing, φ-balancing, and orthogonality-plus-variance losses. Top-1 and top-2 token-choice routing are both included, and expert-choice routing is added as a special case.
For every model we compute three diagnostics. The first is the tie rate: the fraction of inputs whose top-two router probabilities fall within a margin. The second is expert dissimilarity: the distance between competing experts' outputs on the same input, computed on the same per-token expert outputs that the orthogonality loss penalizes. The third is the decomposed explanation variance above.
The route-pinned explainer needs only an intervention that freezes gates during explanation; no new training is required. In deep models the number of configurations grows exponentially with depth. We therefore enumerate configurations only at the layers with the highest routing sensitivity, following VMoER's finding that a small set of layers dominates. All other layers are pinned to their argmax, and the mixture is truncated to configurations covering 95% of router probability mass.
For scale, we apply the same analysis to Granite-MoE (3B) on the multiple-choice question answering suite used by VMoER. Attributions there are computed over prompt tokens.
Expert-choice routing deserves separate treatment. Because each expert picks its tokens from the whole batch, a token's route, and therefore its explanation, depends on the other examples in the batch. We measure this batch-composition multiplicity explicitly, since it is a source of explanation instability that the balance-by-construction literature (Routers in Vision Mixture of Experts) has not considered.
Baselines
Naive reference: plain KernelSHAP. This is KernelSHAP run once with a standard sample budget and reported as a point estimate. It sets the floor for rank churn. We measure churn against a reference distribution: explanations recomputed with many explainer seeds and with retrained model seeds, held out from all methods.
Strongest method for total explanation uncertainty: resampled or conformalized intervals. The main competitor is explainer resampling, meaning bootstrap or repeated SHAP runs with an empirical interval per feature. We also run the two published interval methods, BayLIME posteriors and conformal surrogate intervals. These are the strongest competitors because they target exactly the quantity we bound, without needing any model internals.
Fairness comes from equal budgets. Every method gets the same number of model forward passes per explanation. Our mixture spends that budget across pinned configurations rather than across unpinned resamples. All methods share the same background data and the same tuning budget for sample counts. The interval paper reports valid 95% intervals but no numeric widths or coverage in the file. No reported result was found on Lacuna for rank churn of SHAP on any mixture-of-experts model, so this comparison will establish the first numbers.
To beat these baselines, our intervals must meet nominal coverage of held-out reruns with narrower average width, or with markedly better coverage on near-tie inputs at equal width.
Strongest method for routing stability: VMoER. We apply VMoER's variational router to the same backbones. It reports an ECE reduction from 0.252 to 0.015 on Granite-MoE and 38% higher routing stability under noise (Variational Routing). If its stochastic routing alone removes most of the between-route variance, much of our contribution reduces to "use VMoER". We therefore report explanation churn for VMoER's sampled routes and compare it with our pinned mixture.
Ablation isolating the key ingredient: drop the between-route term. This variant pins every input to its argmax route and reports only within-route uncertainty. If this variant matches the full mixture in coverage, the routing decomposition adds nothing.
Why not just…?
Why not just resample SHAP many times and report the spread? For interval width alone this is a real threat. With unlimited compute it estimates total variance correctly. It falls short in two places. Resampling cannot say why an explanation is unstable, so it gives no basis for intervening, for example by pinning routes or retraining with a different balancing loss. And when perturbations straddle a routing boundary, the resulting distribution is often bimodal, which a symmetric bootstrap interval misrepresents. The equal-budget coverage comparison will settle whether this matters in practice. If bootstrap matches our intervals everywhere, including on tie inputs, we will report that the decomposition is diagnostic only.
Why not just use router entropy as a warning flag? It is cheaper and probably correlates with churn. However, our claim predicts that entropy ignores expert dissimilarity. We test this with a direct control: an MoE whose experts are exact copies of one another. In that model router entropy can be high while the between-route term should be zero. We regress observed rank churn on entropy alone, on dissimilarity alone, and on the between-route term. If entropy predicts churn as well as the between-route term across all four regimes, we will say so and recommend entropy.
Why not just use hard top-1 or expert-choice routing to remove the mixing? Hard routing preserves output calibration under shift (Toward Calibrated MoE Under Distribution Shift), but explainer perturbations still cross routing boundaries, so the explanation jumps instead of blending. Expert-choice routing adds dependence on batch composition. Both are included in the zoo, so the experiments will show whether either eliminates between-route variance.
Why not just use VMoER's Bayesian router? It calibrates outputs, and its routing distribution could be plugged into our mixture as $p(c \mid x)$. We treat it as complementary, not as a substitute. The open question is whether smoothing the router also smooths explanations, or merely makes the jumps random. The VMoER baseline comparison answers this.
Proposed Experiments and Decisive Checks
Synthetic recovery. On the synthetic multi-regime regression, the true per-regime attributions are known. Route-pinned components should recover them, and between-route variance should peak exactly at the regime boundaries. Failure here would falsify the method.
Pinning intervention on real models. On the tabular and CIFAR models, we freeze the router and measure the drop in top-$k$ Jaccard distance and rank-biased overlap against the reference reruns. Claim one predicts that the drop roughly matches the estimated between-route share and concentrates on tie inputs. A drop near zero would refute it.
Regime sweep. Across the four balancing regimes, we plot between-route variance against tie rate × expert dissimilarity. Claim two predicts that this product, not either factor alone, tracks churn. It also predicts that Switch-balanced models show high tie rates with low between-route variance. That combination is the signature of homogenization, and it would also flag IME-style routing explanations as vacuous.
Scale check. On Granite-MoE we repeat the pinning intervention on the multiple-choice question answering suite to test whether routing-induced explanation churn persists at billion-parameter scale. We compare it against VMoER's routing.
A clear positive outcome would show that explanation instability in sparse models is partly an architectural property that anti-collapse training controls. It would also show that it can be reported honestly as a mixture over routes.
References
- Explanation Multiplicity in SHAP: Characterization and Assessment
- Advancing Expert Specialization for Better MoE
- Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers
- Interpretable Mixture of Experts
- Adaptive Conformal Prediction via Mixture-of-Experts Gating Similarity
- Toward Calibrated Mixture-of-Experts Under Distribution Shift
- BayLIME: Bayesian local interpretable model-agnostic explanations
- Approximating Score-based Explanation Techniques Using Conformal Regression
- $\phi$-Balancing for Mixture-of-Experts Training
- Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
- Routers in Vision Mixture of Experts: An Empirical Study