A mixture-of-experts (MoE) network contains many parallel sub-networks, called experts. A small learned module, the router, sends each token to a few of them. These models fail in two familiar ways. In the first, the router sends nearly everything to a handful of experts and the rest starve. In the second, load-balancing losses force traffic to spread evenly, so every expert learns roughly the same function. We argue that both failures are one failure seen from latent-variable modeling. Treat "which expert handles this input" as a hidden variable. Then both failures are posterior collapse: the variational-autoencoder pathology where a model learns to ignore its latent code. Once the choice of expert carries no information about the output, the model also loses its natural source of uncertainty, which is disagreement among experts. That loss links expert collapse to the accuracy–calibration trade-off, meaning the tendency of accurate networks to be overconfident.
The proposal does three things. It measures one quantity, the mutual information between the chosen expert and the predicted output. It tests whether this quantity predicts calibration and error detection better than the load and diversity statistics the MoE literature currently tracks. It then adds a small training-time floor on this quantity, borrowed from the "free bits" fix for posterior collapse in variational autoencoders. The floor keeps experts distinguishable and the router soft enough that sampling routes at test time yields useful uncertainty, without the accuracy loss that comes from squashing all confidences together.
Core Claim
Write a sparse MoE's prediction as a mixture:
$$p(y\mid x)=\sum_e \pi(e\mid x)\,p(y\mid x,e),$$
where $\pi(e\mid x)$ is the router's distribution over expert routes $e$ and $p(y\mid x,e)$ is the network's output when input $x$ takes route $e$. The posterior over routes after seeing an output is $p(e\mid x,y)\propto \pi(e\mid x)\,p(y\mid x,e)$. This posterior equals the router's prior for every $y$ exactly when all routes with nonzero router mass produce the same output distribution, that is, when the route is non-identifiable.
This is the routing analogue of the theorem in Posterior Collapse and Latent Variable Non-identifiability. That paper proves VAE posterior collapse happens if and only if the latent is non-identifiable under the decoder, and it fixes the problem with injective decoders. The size of the gap is
$$I(E;Y\mid X)=\mathbb{E}_x\Big[H\big(p(y\mid x)\big)-\sum_e \pi(e\mid x)\,H\big(p(y\mid x,e)\big)\Big],$$
where $H$ is entropy. This is the same "total minus average entropy" disagreement term used to measure model-based (epistemic) uncertainty in ensembles.
The quantity is zero in both classic failure modes. When experts are homogenized, the routes are non-identifiable. When the router collapses onto a hot expert, $\pi$ is one-hot. So one number covers starvation, homogenization, and the loss of disagreement-based uncertainty.
This explains a tension that Calibrated Knowledge Aggregation in Bayesian Mixture-of-Experts for Continual VQA reports without resolving. That paper averages predictions over experts and finds the calibration gains vanish when expert domains are disjoint and the router sharpens. Meanwhile $\phi$-Balancing for Mixture-of-Experts Training deliberately pursues sharp, disjoint specialization. Our framing predicts that calibration lives in a middle regime that neither line of work targets.
The claim is new in two ways. First, MoE collapse has been analyzed as a load-balancing optimization problem (BASE Layers, φ-balancing), as router geometry (On the Representation Collapse of Sparse Mixture of Experts), or as feature redundancy (SimSMoE), but never as latent identifiability tied to calibration. Second, the theory makes a falsifiable prediction: $I(E;Y\mid X)$, not expert-load Gini or pairwise expert similarity, should track out-of-domain calibration and misclassification detection across training recipes.
Research Program
The first component is measurement. Exact $I(E;Y\mid X)$ over all top-$k$ route combinations in a deep MoE is intractable. We therefore estimate a per-layer version. At layer $l$, we sample a small number of alternative top-$k$ selections from the router's softmax, run the network forward from layer $l$ under each, and compute the entropy decomposition above on the final logits. Summing across layers, or taking the maximum, gives a model-level diagnostic. We validate the estimator on small models where a single MoE layer lets us enumerate routes exactly.
The second component is an explanation of a known failure. Variational Routing, which we call VMoER, adds variational inference over the routing logits of frozen pre-trained MoEs. It reports that its learned-temperature variant suffers "temperature collapse" during training. Our framing predicts this collapse should occur exactly where the backbone's experts are already near non-identifiable. In those regions a stochastic router buys no likelihood, so the optimizer has no reason to keep it stochastic. We test this by logging per-layer $I(E;Y\mid X)$ alongside VMoER's learned temperatures.
The third component is the intervention, a free-bits-style floor:
$$\mathcal{L}=\mathcal{L}_{\text{task}}+\mathcal{L}_{\phi}+\lambda\sum_{l\in S}\max\big(0,\;\delta-\hat I_l\big).$$
Here $\mathcal{L}_{\phi}$ is the population-level entropy balancing term from φ-balancing, which we keep unchanged to prevent starvation. $S$ is one randomly chosen MoE layer per step, which keeps the extra cost to one partial forward pass. $\hat I_l$ is the estimated route–output mutual information at that layer. $\delta$ is the floor and $\lambda$ its weight.
Raising disagreement on training examples would fight the task loss. We therefore compute $\hat I_l$ on a small held-out stream of training-distribution data, the way a calibration method uses a validation split, but as a plain penalty rather than bilevel optimization. Because it is a floor and not a maximization, the term goes inactive once experts stay distinguishable. This is the mechanism by which we hope to avoid the "flat entropy penalty hurts accuracy" result reported by Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration, which calls its method CALM.
At inference, we sample routes several times, analogous to dropout at test time, and average the predictive distributions. A single deterministic pass remains available when cost matters.
The proposal deliberately does not adopt the injective Brenier-map decoders from the identifiability paper. Their per-forward gradient computation is too expensive for transformer experts, and identifiability here only needs to hold where the router places mass. The floor is a cheap, local stand-in for that structural guarantee.
Baselines
The naive reference is a standard top-$k$ MoE trained with a Switch-style auxiliary balancing loss, evaluated with a single deterministic pass. Any claim about collapse or calibration must first beat this.
The strongest training-time collapse baseline is φ-balancing. It is the best-performing balancing method in the file: on Moonlight-16B-A3B it raises Math500 from 64.8% to 66.2% and HumanEval from 73.2% to 75.9% over standard balancing. It reports no calibration numbers. We run it with identical architectures, data, token budgets, and the same number of hyperparameter trials as our method. We give it post-hoc temperature scaling and Monte Carlo route sampling at inference, so the only difference from our method is the floor.
The strongest calibration baseline for MoEs is VMoER. On Granite-MoE (3B) it reports ECE falling from 0.252 to 0.015 on multiple-choice question answering, with a 12% average gain in out-of-domain AUROC (the area under the curve for separating in-domain from out-of-domain inputs). ECE, expected calibration error, measures the gap between stated confidence and actual accuracy. We run VMoER's mean-field and full-covariance logit variants on the same backbone, using the same router-only parameter budget our fine-tuning uses.
Beating VMoER does not require a lower in-domain ECE than 0.015, which leaves little room. It requires matching ECE within seed noise while improving misclassification-detection AUROC and out-of-domain ECE. The latter two are where the accuracy–calibration trade-off actually bites.
The diversity baseline is SimSMoE's CKA penalty combined with φ-balancing and route sampling. CKA, centered kernel alignment, is a measure of how similar two sets of representations are, and SimSMoE penalizes it to push expert features apart. On calibration, the file has no reported result on Lacuna for SimSMoE. Its reported numbers are 1.08 versus 1.11 bits per character on enwik8 and a 4.8% accuracy gain on BANKING77.
The key ablation removes the floor while keeping everything else, including route sampling at inference. This isolates whether training for identifiable routes matters, or whether merely sampling routes from an ordinary model already supplies the uncertainty.
Why not just…?
Why not just apply temperature scaling to a well-balanced MoE? This is a real threat in-domain. A single fitted temperature can drive in-domain ECE very low on almost any model. However, CALM shows a validation-fitted temperature does not transfer out of domain: its training-time method cuts conf-ECE by 41.1% relative to the baseline on Llama-3.1. A single temperature also rescales all confidences monotonically, so it cannot change which examples are ranked as likely errors. Our misclassification-AUROC and out-of-domain ECE comparisons against φ-balancing plus temperature scaling decide this. If temperature scaling matches us on both, the method claim fails, though the diagnostic claim may survive.
Why not just combine SimSMoE's diversity penalty with φ-balancing? This is the most serious alternative, because it already pushes experts apart and keeps them loaded. Our objection is that CKA diversifies internal features everywhere, without regard to whether output distributions differ. Pushed hard, it should produce exactly the sharp, disjoint specialization that the continual-VQA paper reports kills disagreement. That is a testable prediction: sweeping the CKA weight should trace an inverted-U in $I(E;Y\mid X)$ and in out-of-domain calibration, whereas the floor targets the output-level quantity directly. If CKA plus balancing reaches the same $I(E;Y\mid X)$ and calibration at equal cost, our method adds nothing, and we would report the measurement as the contribution.
Why not just use a deep ensemble or VMoER-style post-hoc variational routing? Ensembles of dense models also produce disagreement, but at multiples of training compute. We compare against an ensemble matched in total training FLOPs. VMoER is cheaper (about 1.07% extra FLOPs) but works on frozen experts. If those experts are already homogenized, our theory says no router posterior can recover disagreement. The temperature-collapse logging tests this directly. VMoER on a floor-trained backbone versus a standard backbone should show whether the remedy belongs in training or after it.
Why not just use focal loss? Calibrating Deep Neural Networks using Focal Loss limits logit growth in monolithic networks, and we include it as an orthogonal add-on. It changes the loss on outputs but gives the router no reason to keep routes distinguishable, so we expect it to be additive with the floor rather than a replacement.
Proposed Experiments
The first stage is a controlled from-scratch study. We train small MoE vision transformers on CIFAR-10 and Tiny-ImageNet, the suites used in the focal-loss paper, which reports no numbers on Lacuna that we can quote. We also train small MoE language models on enwik8 and WikiText-103 and fine-tune them on BANKING77 and SST-2. Recipes span standard balancing, φ-balancing, SimSMoE at several weights, expert-choice routing, and our floor at several $\delta$ values, with three or more seeds each.
The decisive check is a cross-model correlation. Across all of these trained models, does held-out $I(E;Y\mid X)$ predict misclassification AUROC and out-of-domain ECE better than expert-load Gini or mean pairwise CKA? Out-of-domain here means a held-out class split, or for language models, the shift from WikiText-103 to SST-2 text. If load or CKA predicts as well or better, the unifying claim is wrong.
The second stage is pre-trained MoE fine-tuning. We fine-tune only the routers of Granite-MoE (3B) on the multiple-choice suite used by VMoER: in-domain OpenBookQA, ARC-Challenge, and SciQ, and out-of-domain MMLU-Law and MedMCQA. We report accuracy, ECE, Brier score, misclassification AUROC, and out-of-domain AUROC, against VMoER's 0.252→0.015 ECE result. Here we also run the temperature-collapse test.
The third stage targets the specialization–calibration tension directly on continual VQA-v2. The Bayesian-aggregation baseline reports 64.16% accuracy and ECE reduced from 0.15 to 0.07. We ask whether the floor preserves its calibration gains on the TDIUC visual-domain streams, where expert domains are more disjoint and that method predicts its own degradation.
Across all three stages, success has one shape. The floor must match φ-balancing's accuracy within seed noise while improving out-of-domain ECE and error-detection AUROC beyond temperature scaling. Its cost should stay at a single extra partial forward pass per step.
References
- Posterior Collapse and Latent Variable Non-identifiability
- Calibrated Knowledge Aggregation in Bayesian Mixture-of-Experts for Continual VQA
- $\phi$-Balancing for Mixture-of-Experts Training
- BASE Layers: Simplifying Training of Large, Sparse Models
- On the Representation Collapse of Sparse Mixture of Experts
- SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse
- Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers
- Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
- Calibrating Deep Neural Networks using Focal Loss.