A mixture-of-experts model is a neural network whose layers contain many parallel sub-networks ("experts") plus a small gating network (the "router") that decides which few experts process each token. Recent work treats the router's decision as a random latent variable and fits an approximate posterior over it with variational inference. Doing so improves calibration and out-of-distribution detection in pretrained language models at almost no cost. Every one of these methods, however, assumes that routing uncertainty is independent from one token to the next. This is the mean-field assumption: the approximate posterior factorizes into a product of independent per-token pieces. We argue that this assumption does specific, predictable damage. When a model reads an unfamiliar passage, its uncertainty about where a span of tokens should go is shared across the whole span. Independent per-token noise instead averages out as the sequence gets longer. The result is that a mean-field router's sequence-level uncertainty signal shrinks roughly as one over the sequence length, which is exactly the classic variance-underestimation failure of mean-field methods, now appearing along the time axis.
We propose a chain-structured variational router. Each token's routing-logit noise is correlated with the previous token's, and the correlation strength is predicted from the two adjacent hidden states, so it can approach one inside a coherent span and drop to zero at a boundary. The posterior stays reparameterizable, takes linear time in sequence length, is guaranteed to remain a valid covariance during training, and conditions only on past tokens, so it works during token-by-token generation. The core claim is testable. Mean-field routing's out-of-distribution detection should degrade with prompt length, correlated routing's should not, and the difference should show up most clearly on long open-ended generation, where no variational router has yet been evaluated.
What Exists and Where the Gap Is
Three recent papers make the router probabilistic, and all three keep tokens independent. Variational Routing (VMoER) freezes a pretrained backbone and learns a small residual head that outputs a Gaussian over router logits, in two versions. VGLR-MF is diagonal across experts. VGLR-FC uses a full covariance across experts. The paper reports that calibration error on Granite-MoE 3B falls from 0.252 to 0.015 and out-of-distribution AUROC rises by about 12% on average, with +1.07% FLOPs. Its posterior is nonetheless a product over tokens, and it was evaluated only on multiple-choice question answering. DirMoE recasts routing as a Dirichlet variational autoencoder with spike-and-slab gates. ProbMoE samples exact size-$k$ expert subsets with a differentiable estimator. Both target training-time routing quality rather than uncertainty, and both treat tokens as independent samples. Mean Field Equilibria for MoE couples tokens only through aggregate expert load in a congestion game. That is the physics sense of "mean field." It infers no posterior, so it does not touch the problem we target.
The motivating evidence comes from both directions. Opper et al. show that factorized approximations underestimate posterior variance because they cannot represent co-movement among latents. StableMoE shows that routing for nearly 30% of tokens is still fluctuating halfway through training, a sign that per-token assignments are noisy in ways the model does not track coherently. The method comes from instance-level structured variational inference. HoT-VI parameterizes overlapping local marginals through conditional correlations, which guarantees positive-definiteness at cost linear in the dependency order. Copula-SVI extends instance correlations with vine copulas. HoT-VI's stated limitation is that its dependency backbone is fixed in advance. Our router keeps a fixed chain topology but makes the edge strengths input-dependent, so a near-zero correlation effectively cuts the chain wherever the content changes. Neither paper has been applied to discrete or sparse routing.
Core Claim
Let $\mathbf{g}_t \in \mathbb{R}^E$ be the router logits for token $t$ over $E$ experts. Let $\boldsymbol{\mu}_t$ and $\boldsymbol{\sigma}_t$ be the per-token mean and scale from a VMoER-style residual head. Let $\boldsymbol{\epsilon}_t$ be a standard normal vector, and let $\rho_t \in (-1,1)$ be a scalar predicted by a small network from the hidden states at positions $t-1$ and $t$. We define the noise recursively:
$$ \boldsymbol{\eta}_t = \rho_t\, \boldsymbol{\eta}_{t-1} + \sqrt{1-\rho_t^2}\;\boldsymbol{\epsilon}_t, \qquad \mathbf{g}_t = \boldsymbol{\mu}_t + \boldsymbol{\sigma}_t \odot \boldsymbol{\eta}_t . $$
Each token's marginal is the same Gaussian the mean-field router would use. Only the joint distribution across tokens changes. Setting $\rho_t = 0$ recovers VGLR-MF exactly, and setting $\rho_t = 1$ recovers a single shared noise draw per sequence. Because the recursion is causal, it runs during autoregressive decoding. In that respect it differs from Expert Choice routing (Zhou et al.), which needs global batch access.
The quantitative prediction follows from the variance of a sequence average. For any sequence-level statistic that averages per-token routing effects, the variance under independent noise is about $\sigma^2/T$. Under a stationary chain with correlation $\rho$, it is about $\sigma^2(1+\rho)/((1-\rho)T)$, the standard result for the mean of a first-order autoregressive process. The mean-field router's uncertainty is therefore biased toward overconfidence by a factor that grows with how coherent the true routing uncertainty is.
The extension we would add only if the chain alone works is a separable covariance. This would be the token chain in one direction and VGLR-FC's expert covariance in the other, implemented by replacing $\boldsymbol{\epsilon}_t$ with $\mathbf{L}\boldsymbol{\epsilon}_t$, where $\mathbf{L}$ is a Cholesky factor of the expert covariance.
Research Program
We first diagnose the problem before building anything. On a frozen MoE with a VMoER mean-field head, we measure how the out-of-distribution AUROC from routing uncertainty changes with prompt length. We also measure how strongly the posterior-mean routing of adjacent tokens agrees, a proxy for how much coherence a structured posterior could exploit. If the mean-field AUROC does not fall with length, the core motivation fails and we stop.
The second step builds the chain router. It uses the same residual-head placement as VMoER, which applies the head to its top 10 most sensitive layers and keeps the backbone frozen. The ELBO keeps VMoER's per-token likelihood terms. The KL term is computed against a matching chain prior, which has a closed form for Gauss-Markov chains. Uncertainty at test time is computed by sampling routing trajectories, and we report predictive entropy and mutual information.
The third step evaluates in a regime prior work skipped, which is open-ended generation. There, sampled trajectories produce whole continuations, and we score uncertainty by agreement across sampled answers. Cross-layer coupling, in the spirit of structured deep Gaussian processes, is a natural next axis, but it is out of scope here.
Baselines
The naive reference point is the frozen pretrained model with its deterministic router, plus temperature scaling of output logits fit on a held-out in-distribution split. VMoER reports a calibration error of 0.252 for the unscaled Granite-MoE 3B router. Temperature scaling will make calibration error trivially small, so it serves mainly to show that calibration error alone cannot separate the methods. Out-of-distribution AUROC and selective-prediction curves can.
The direct ablation of our key ingredient is VGLR-MF from VMoER. Our model reduces to it when $\rho_t = 0$, so any difference is attributable to cross-token correlation. Both models will use the same layers, head width, training data, number of optimization steps, and hyperparameter search budget, and both will use the same number of Monte Carlo samples at test time.
The strongest existing competitor is VGLR-FC, also from VMoER. It models correlation across experts, not tokens, and the paper reports the best result among its routers: calibration error 0.015 on Granite-MoE 3B and roughly 12% average AUROC gain on MMLU-Law and MedMCQA. We will reimplement it under the same budget. Beating it means higher out-of-distribution AUROC on MMLU-Law and MedMCQA overall, with the gap widening in the longest prompt-length tercile, and calibration error no worse than VGLR-FC's. VMoER reports no numbers for open-ended generation, so on GSM8K we compare against our own reruns.
A second ablation fixes $\rho_t = 1$, giving one shared noise draw per sequence, in the spirit of hierarchical variational models, which place a prior over the parameters of a factorized distribution. A third fixes $\rho_t$ to a single learned constant. Together these isolate whether input-dependent correlation matters or whether any shared noise suffices.
Why not just…?
Why not just rely on the router's input already being contextual? The router reads hidden states produced by attention, so the mean routing of each token already depends on the whole sequence. That is true, and it is why we leave the means untouched. What contextual features cannot provide is correlation in the sampled noise. Under mean-field sampling, two adjacent tokens with identical uncertainty still flip experts independently, so sequence-level spread averages away. The $\rho_t = 0$ ablation tests this objection directly. If it matches our model on length-stratified AUROC, the objection wins.
Why not just share one noise draw per sequence? This is the most serious threat. It is simpler, and for short multiple-choice prompts it may capture most of the benefit. We include it as an ablation. Our prediction is that it overstates coherence when a sequence mixes content types, such as a law question with an embedded arithmetic step, and that it will lose to the input-dependent chain on long generations and mixed-domain prompts. If it ties everywhere, we will report that the useful structure is hierarchical rather than sequential and recommend the simpler model.
Why not just model correlation across experts, as VGLR-FC does? VGLR-FC already captures much of the calibration gain at small cost. But expert covariance is a statement about one token, and it cannot stop independent per-token noise from averaging out over length. The two kinds of correlation are complementary. The separable extension tests whether they combine, and the length-stratified comparison tests whether ours adds anything VGLR-FC lacks.
Why not just penalize routing changes between adjacent tokens, or freeze the router as StableMoE does? A smoothness penalty or a frozen lookup router (StableMoE) makes the point estimate of routing coherent but produces no uncertainty at all. It therefore offers no out-of-distribution score or selective-prediction signal. We will still report routing run-lengths and stability under input noise, where VMoER reports a +38% stability gain, to check that our posterior means are not less stable than these deterministic alternatives.
Proposed Experiments
The backbones are Granite-MoE 3B and DeepSeek-MoE-16B, matching VMoER. For in-distribution question answering we use ARC, OpenBookQA, and SciQ, with MMLU-Law and MedMCQA as out-of-distribution sets. We report calibration error, AUROC, and routing stability under noise, all stratified by prompt-length tercile. For open-ended generation we use GSM8K with sampled reasoning chains on the same backbones, and also on OLMoE-1B-7B, where ProbMoE reports 50.19% accuracy against 45.94% for conventional routing. On GSM8K we report accuracy, which should be unchanged because the means are shared, and area under the risk-coverage curve when abstaining on the most uncertain problems. We also report the learned $\rho_t$ profiles. These should be high within numbers and expressions and low at sentence boundaries, if the router is learning real structure rather than a constant. Overhead in FLOPs and memory is measured against VGLR-FC's reported +1.07% and +1.15%.
The decisive check is a single plot of out-of-distribution AUROC against prompt length. Mean-field should slope downward, our router should stay flat, and the gap at the longest tercile should exceed seed variance across five seeds. The second decisive check is whether the input-dependent chain beats the shared-noise ablation on GSM8K selective prediction. If neither holds, cross-token correlation is not worth modeling in routers, and that negative result would itself settle an open question that VMoER, DirMoE, and ProbMoE all left implicit.
References
- Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers
- DirMoE: Dirichlet-Routed Mixture of Experts
- ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
- Bridging Game Theory and Transformer Routing: Mean Field Equilibria for Mixture of Experts
- Variational Linear Response
- StableMoE: Stable Routing Strategy for Mixture of Experts
- HoT-VI: Reparameterizable Variational Inference for Capturing Instance-Level High-Order Correlations
- Copula-SVI: Vine-Copula Variational Inference with Stein Refining for Instance-Level Correlation Capturing
- Mixture-of-Experts with Expert Choice Routing
- Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties
- Hierarchical Variational Models