Recovering Cross-Expert Routing Correlations Without Training Them

Mixture-of-experts language models send each token to a few specialist sub-networks ("experts") chosen by a small gating network called the router. When the router is confident and wrong, the whole prediction is confidently wrong. Recent work fixes this by making the router Bayesian: it outputs a probability distribution over its own scores for each expert rather than a single score. The version that also models how those scores co-vary across experts calibrates well, but it must learn a full covariance matrix whose size grows with the square of the number of experts.

We propose a different route. Train only the cheap version, which assigns each expert an independent variance. Then recover the missing cross-expert correlations after training, analytically, from the curvature of the training objective. The tool is linear response, a correction from Bayesian statistics. It reads off the true covariance from how the fitted means would shift if the objective were nudged slightly. If this works, calibrated routing for fine-grained models with many experts costs almost nothing beyond a mean-field router and a one-time offline computation.

Core Claim

The closest existing work is VMoER (Variational Routing). It places a Gaussian distribution over router logits in the most sensitive layers of frozen MoE backbones (Granite-MoE, Qwen-MoE, DeepSeek-MoE) and compares two forms. In the mean-field form (VGLR-MF), each expert's logit has its own variance and the logits are treated as independent. In the full-covariance form (VGLR-FC), an explicit covariance matrix is learned per token. VMoER reports Granite-MoE expected calibration error falling from 0.252 to 0.015 and out-of-distribution AUROC improving by 12% on average. The file does not attribute these numbers to a specific variant.

Linear Response Variational Bayes (Giordano et al. 2015) shows that a mean-field fit, which factorizes the approximate posterior into independent pieces, systematically misses correlations. That paper also shows the correlations can be recovered post hoc:

$$\hat\Sigma_{LR} = (I - VH)^{-1}V .$$

Here $V$ is the diagonal mean-field covariance and $H$ is the Hessian of the expected log joint density with respect to the variational means. For Gaussian targets this formula is exact.

Our claim is that applying this correction to a mean-field variational router, VGLR-MF, recovers the calibration and robustness gains of VGLR-FC. It should match VGLR-FC at small expert counts. At large expert counts, where learning a full covariance is costly and unstable, it should beat or match VGLR-FC at a fraction of the training parameters. VMoER never attempts a post-hoc correction. LRVB has never been applied to routing in neural MoE models, and its authors list mixture structures with discrete assignments as an open extension.

The mechanism has one non-obvious ingredient that also separates it from a plain Laplace approximation, which places a Gaussian at the router's point estimate using local curvature. Top-$k$ selection makes the loss non-smooth in the logits. The curvature at a single logit vector therefore says nothing about what happens when the selected set changes. The linear response Hessian is taken of the expected objective under the mean-field distribution, which smooths over selection changes. We make that expectation deterministic using the fixed-draw idea of DADVI (Giordano et al. 2023), so its Hessian can be computed by automatic differentiation.

Research Program

Stage one: train the mean-field router. We reproduce VGLR-MF on the frozen VMoER backbones in the same layers, with the same residual inference heads.

Stage two: compute per-token corrections offline. On a held-out labelled calibration split, we treat each token's logit vector in each corrected layer as a small local latent variable. We draw $N=30$ fixed standard-normal samples per token, the DADVI default. Giordano et al. argue that this suffices when the per-sample latent dimension is small, because the inexpressive mean-field family cannot overfit fixed draws.

Using these draws, we form the deterministic expected objective and compute $H$ only on the "active block". This block is the set of experts that enters the top-$k$ under at least one of the draws; all other experts keep their mean-field variance. This bounds the inverse to a small matrix even when the layer has very many experts. It also makes cost scale with routing ambiguity rather than with pool size.

Within the active block, a fixed-draw top-$k$ objective is still piecewise constant in selection. We therefore compute $H$ two ways and report both. The first uses the dense softmax relaxation of the gate. The second uses a Stein-identity estimator, which expresses derivatives of the expectation as weighted averages of the objective over the fixed draws and needs no derivative through the selection, with control variates.

Handling biased means. LRVB is only as good as the mean it linearizes around. Mean-field means are biased under strong dependence, and routing collapse is exactly such a bias (S2MoE; MoLE). We therefore take a few trust-region Newton steps on each token's deterministic objective, starting from the amortized mean, before forming $H$. We record how far the mean moves, which gives a direct diagnostic of the failure mode Giordano et al. warn about.

Stage three: deploy without labels. Per-token corrections need labels, so they cannot be computed at test time. We compress them into a per-layer correlation matrix $R_\ell$, optionally one per cluster of router-mean vectors, pooled over calibration tokens. At test time the token's own mean-field standard deviations are combined with the stored correlation:

$$\Sigma_t = D_t^{1/2} R_\ell D_t^{1/2},$$

where $D_t$ is the diagonal of the token's mean-field covariance. This adds one stored $K\times K$ matrix per corrected layer and no extra trainable parameters per token. As an extension, a small head can be distilled to predict low-rank per-token corrections, which tests whether correlation structure is genuinely input-dependent.

Baselines

The naive reference is the deterministic pretrained router with no uncertainty. VMoER reports its expected calibration error on Granite-MoE as 0.252. We rerun it on our splits rather than relying on the reported value.

The strongest existing method is VGLR-FC from Variational Routing. It is the strongest competitor because it targets exactly the correlations we recover, on the same models and benchmarks. The figure to approach is the VMoER calibration error of 0.015 on Granite-MoE, with the caveat that the file does not say which variant produced it. We train VGLR-FC with the same layers, data and number of optimizer steps as VGLR-MF, and give both the same hyperparameter search budget. Beating it means matching its calibration error and AUROC within seed variance at small $K$. At large $K$ it means matching or exceeding it with fewer trainable parameters and lower training memory.

The uncorrected VGLR-MF is the ablation that isolates linear response. Any gain over it is attributable to the correction alone, because the trained network is identical.

Two further ablations isolate the specific ingredients. The first is a Laplace correction at the router mean: the same Hessian, but without the mean-field variances and without the expectation. This tests whether the smoothing over selection changes matters. The second is an empirical correlation $R_\ell$ estimated from co-activation statistics of the mean-field router on the same calibration split. This tests whether curvature adds anything beyond counting which experts fire together.

We also run output-level model averaging over the top-$k$ experts, in the style of Mozaffari et al., and post-hoc temperature scaling of the final output. Both are fit on the same calibration split.

Why not just…?

Why not just train the full-covariance router, since it adds only about 1% compute? VMoER reports VGLR-FC adding 1.07% FLOPs and 1.15% memory, and at small expert counts that is a real threat. The proposal must therefore win on scaling and stability, not on small models. Parameter count for a per-token full covariance grows quadratically in $K$. MoLE already shows routing degrading as the pool grows to 128 LoRA experts (Wu et al.). Giordano et al. (2023) find that expressive families overfit fixed-sample objectives in high dimension. Our expert-count sweep settles this. If VGLR-FC remains cheap and stable at 128 experts, the cost argument fails, and the contribution reduces to "same quality, no retraining".

Why not just temperature-scale the output? This is a strong baseline for in-distribution calibration error and may tie us there. It cannot change which experts are selected. So it should not improve routing stability under input noise, which VMoER measures and reports a 38% gain on. It also should not help out-of-distribution AUROC on MMLU-Law and MedMCQA. Those two metrics are where we expect separation, and we report all three.

Why not just apply a Laplace approximation, either to the router or to all weights? Whole-network Laplace in sparse information form ( Estimating Model Uncertainty of Neural Networks in Sparse Information Form) still needs about 105 MB for ResNet-50 and linearizes around a single mode. Laplace on router logits is cheap, but its curvature ignores top-$k$ selection flips. The router-Laplace ablation measures this directly. If it matches linear response, the smoothing argument is wrong and we will say so.

Why not just average expert outputs? Mozaffari et al. report calibration error falling from 0.15 to 0.07 on VQA-v2 through disagreement pooling. They also note that the benefit vanishes when routing sharpens on distinct inputs. We test both methods on the same out-of-distribution splits.

Proposed Experiments

We use the VMoER suite: Granite-MoE (3B), Qwen-MoE (2.7B) and DeepSeek-MoE (16B). In-distribution evaluation uses OpenBookQA, ARC-Challenge and SciQ; out-of-distribution evaluation uses MMLU-Law and MedMCQA. We measure accuracy, calibration error, out-of-distribution AUROC, and routing stability under input noise. On the compute side we measure training parameters, FLOPs, memory, and offline correction time.

The expert-count sweep composes LoRA experts in MoLE from small pools up to 128 and evaluates on Big-Bench Hard. MoLE reports a 3.0-point margin over LoRAHub at 128 experts, which serves as the accuracy floor.

The decisive validity check is small-scale. For a single router layer and a few hundred tokens, the per-token logit posterior is low-dimensional enough to sample with MCMC. We compare off-diagonal covariance error for four estimates: mean-field, linear response with the softmax-relaxed Hessian, linear response with the Stein estimator, and the empirical co-activation correlation. This mirrors how LRVB was validated against Gibbs sampling. We also stratify the error by how far the Newton refinement moved the mean, to test whether collapsed routers break the correction.

The claim fails under any of three outcomes. It fails if linear response does not beat uncorrected mean-field on AUROC or routing stability. It fails if the co-activation or router-Laplace ablation matches linear response. It fails if VGLR-FC stays cheap and stable throughout the sweep, because then the case for a post-hoc correction rests only on avoiding retraining.