A sparse mixture-of-experts model is a network in which a small learned "router" sends each token to only a few of many sub-networks, called experts. These models routinely suffer expert collapse: a few experts win early, receive most tokens, and improve further, while the rest stop learning. The usual fixes adjust load directly, through penalties, bias terms, or assignment algorithms, so that every expert gets work. None of them asks whether the router is trained toward the right target. A parallel literature on learning to defer studies a classifier that decides which of several external experts should answer each input. That work proves which training losses provably converge to the best possible routing decision, a property called classification calibration or consistency.
This note proposes treating the internal router of a mixture-of-experts layer as a deferral classifier. The router would be supervised with cheap estimates of how well every expert would have handled a token, including experts it did not pick, and trained with a loss that is provably consistent for the best routing under a capacity limit. The thesis is that collapse is partly a feedback problem. A standard router only observes the experts it already chose, so its training signal is biased toward incumbents. Giving it occasional full-information, counterfactual labels, inside a loss with known calibration properties, should yield both balance and real specialization without hand-tuned penalties.
Core Claim
In a standard top-$k$ router, the gradient for expert $j$'s gate exists only when $j$ was selected. The router is effectively learning from bandit feedback on $k$ of $E$ arms, with no exploration. That is the rich-get-richer loop described in Scaling Vision with Sparse Mixture of Experts. Learning-to-defer theory says what the router should be trained toward when competence labels for all experts are observed. Consistent Estimators for Learning to Defer to an Expert casts deferral as an augmented classification problem with a Fisher-consistent cross-entropy surrogate. Why Ask One When You Can Ask $k$? extends this to consistent top-$k$ committee selection.
We claim three things. First, such labels can be produced cheaply for internal experts with a first-order counterfactual. Second, combining those labels with capacity prices defines a well-posed "capacity-constrained Bayes router." Third, training the router toward it with a rank-consistent loss reduces dead experts and routing regret at matched compute, beyond what load-balancing alone achieves.
The mechanism has two parts. For a probe token $x$ at a mixture layer, let $y(x)$ be the layer's current mixture output and $f_j(x)$ the output of expert $j$. Let $g(x)$ be the gradient of the training loss with respect to the layer output, which the backward pass already computes. The estimated loss change from substituting expert $j$ is
$$\Delta_j(x) = g(x)^\top \big(f_j(x) - y(x)\big),$$
which is exactly the quantity the standard router gradient uses, but evaluated for all experts rather than only the selected ones.
The router, with logits $r(x)$, is trained with cross-entropy against the soft target
$$q_j(x) = \mathrm{softmax}_j\!\left(-\frac{\Delta_j(x) + \lambda_j}{\tau}\right).$$
Here $\lambda_j$ is a capacity price for expert $j$, updated by dual ascent on an exponential moving average of its load, as in $\phi$-Balancing for Mixture-of-Experts Training. The temperature $\tau$ controls how sharply the target concentrates.
By Lagrangian duality, the population-optimal assignment under per-expert capacity limits is the top-$k$ of $-(\Delta_j + \lambda^*_j)$ at the optimal prices $\lambda^*$. With an unrestricted router, cross-entropy against $q$ is minimized at $\mathrm{softmax}(r) = q$, which preserves that ranking. This is the rank-consistency property shown for softmax cross-entropy in On the consistency of top-k surrogate losses. Capacity therefore enters the target rather than appearing as a penalty that pulls against it. That targets the open question of whether consistency survives load constraints, which Mastering Multiple-Expert Routing leaves unaddressed: its consistent routers will happily send every input to one dominant expert.
Relation to Prior Work
The closest work is When More Experts Hurt. It shows that multi-expert deferral losses which sum expert correctness inject label noise growing with the number of experts. It fixes this with PiCCE, which supervises only the most confident among the correct experts. That paper motivates our temperature study: the hard limit $\tau \to 0$ is a PiCCE-like target, and large $\tau$ is the flattened, noisy aggregate PiCCE warns against. It also differs from us in three ways. It assumes fixed, fully observed external experts. Its experts make class predictions. It uses at most tens of experts. Ours co-adapt with the router, emit hidden vectors, number 64 or more, and are observed only on sampled probe tokens.
Why Ask One When You Can Ask $k$? and Mastering Multiple-Expert Routing supply consistent top-$k$ and hypothesis-class-consistent deferral losses. They also treat experts as static black boxes with no capacity limits, which is exactly the setting in which consistent routing causes collapse.
On the mixture-of-experts side, three methods rebalance load without changing what the router learns to prefer. Auxiliary-Loss-Free Load Balancing uses heuristic bias feedback. $\phi$-Balancing uses population-level dual prices, and we reuse its price update. BASE Layers solves an exact assignment that it cannot reproduce at inference.
ProbMoE makes top-$k$ selection differentiable via marginal probabilities, but gives no calibration analysis. Its gradients still flow only through expert outputs actually computed on the current tokens.
Coupling Experts and Routers diagnoses the same disease we do: routers do not know what experts can compute. It treats it with a contrastive loss on synthetic proxy tokens. We instead give the router measured competence on real tokens, with a consistency argument.
Variational Routing improves empirical calibration of model outputs but proves nothing about the router's surrogate.
Statistical Perspective of Top-K Sparse Softmax Gating constrains our expectations. With more experts than true components, gating parameters converge very slowly. We therefore measure routing regret and functional specialization rather than hoping to recover "true" experts.
Finally, On the consistency of top-k surrogate losses also shows that cross-entropy can lose top-$k$ consistency when restricted to linear models, while a modified hinge $\psi_5$ is top-$k$ calibrated. Mixture-of-experts routers are typically linear in the hidden state. This makes the comparison between cross-entropy and $\psi_5$ a real empirical question here rather than a curiosity.
Research Program
The work has four strands: a theoretical result, a validation of the counterfactual estimate, a diagnostic that tests the core claim on existing methods, and the method itself.
Theory. We will state and prove the constrained-consistency result in the realizable and unrestricted settings: rank consistency of price-shifted cross-entropy for the capacity-constrained Bayes router. We will then characterize when it fails for linear routers and whether $\psi_5$ restores it.
Validating the counterfactual. Before relying on $\Delta_j$, we will compare it against exact counterfactuals on small models. An exact counterfactual swaps expert $j$ into the layer and reruns the rest of the network. The comparison metric is rank correlation, and in particular agreement on the top-$k$ set. If the first-order estimate ranks experts poorly in deep layers, we fall back to exact swaps on fewer probe tokens. We will report the resulting cost honestly.
Diagnostic. In standard, loss-free-balanced, and $\phi$-balanced models, we will measure routing regret on probe tokens throughout training. Routing regret is the gap between the loss of the router's top-$k$ and the counterfactual-best top-$k$. We will also test whether experts that went cold would in fact have been competent. If cold experts are uniformly incompetent, then the feedback-bias story is wrong and the proposal loses its rationale. This diagnostic is therefore a go/no-go gate.
The method. Each step, a fraction $\rho$ of tokens is probed. Each probe token is evaluated on a random subset of $m$ experts, not all $E$, and the target is restricted to that subset with importance correction. The router receives the consistent loss on probe tokens and its usual gradient elsewhere.
Baselines
The naive reference point is a Switch-style top-$k$ router with an auxiliary load-balancing penalty. This is the default in practice and the configuration in which homogenization is documented (Advancing Expert Specialization for Better MoE).
The strongest load-balancing competitors are loss-free balancing and $\phi$-balancing. Both fix balance without changing the routing target, so beating them isolates our contribution. On a 1B-parameter model, loss-free balancing reports validation perplexity 9.50 against 9.56 for the auxiliary-loss baseline, and a maximum load violation of 0.04 against 0.72 (Auxiliary-Loss-Free Load Balancing). Their corpus is not available to us, so we rerun both methods on our data.
The strongest router-learning competitor is ProbMoE. On OLMoE-1B-7B it reports GSM8K accuracy of 50.19% against 45.94% for conventional routing (ProbMoE). That number is our fine-tuning target. Expert-router coupling is included as the other method that explicitly tries to inform the router about expert ability. No standalone task score is reported in the file for it; no reported result was found on Lacuna for coupling on GSM8K.
Fairness works as follows. All methods share data, architecture, expert count, and a fixed hyperparameter-search budget. Our probing cost, of order $\rho m / k$ times the expert-layer compute, is matched by giving every baseline additional training tokens of equal FLOPs.
Three ablations isolate the key ingredient:
- Probing without consistency. Run the same probing with random routing on probe tokens and the usual gradient only, which supplies exploration without a calibrated target.
- Consistency without capacity. Use the consistent target with $\lambda = 0$. This should reproduce collapse onto incumbents and confirms that capacity belongs in the target.
- Surrogate and temperature. Swap cross-entropy for $\psi_5$, and sweep $\tau$ from PiCCE-like hard targets to flat ones.
Beating the baselines means lower perplexity and lower routing regret at matched FLOPs, with load violation no worse than loss-free balancing. In fine-tuning it means clearing 50.19% on GSM8K on the same backbone.
Why not just…?
Why not just use loss-free balancing or $\phi$-balancing, which already solve load imbalance? They do solve balance, and this is a real threat on perplexity. They can only redistribute tokens among experts whose quality the router has already, and biasedly, estimated. If the diagnostic shows near-zero routing regret under $\phi$-balancing, our method has nothing to add. In that case we would report that outcome rather than the method. The contrast experiment is regret and specialization at equal load violation.
Why not just add exploration noise to the router? Random exploration gives cold experts gradient and is nearly free. Probing without the consistent target is exactly this alternative, run at identical cost. If it matches the full method, the honest conclusion is that collapse is an exploration problem and the calibration machinery is unnecessary.
Why not just use ProbMoE's differentiable top-$k$? Its marginal-probability gradients still depend only on expert outputs computed for the sampled subset. It also has no guarantee that its minimizer selects the best experts. The head-to-head comparison is on OLMoE-1B-7B fine-tuning, and we will additionally measure ProbMoE's routing regret on probe tokens.
Why not just use Expert Choice routing, which is balanced by construction? Experts selecting tokens can leave hard tokens unprocessed, and it cannot run causally during token-by-token generation (Routers in Vision Mixture of Experts). It is included in the vision-free pre-training comparison only for completeness, with token-drop rate reported.
Proposed Experiments
Synthetic test. Gaussian mixtures of linear experts with a known number of components, deliberately over-specified, in the regime of Statistical Perspective of Top-K Sparse Softmax Gating. Here exact counterfactuals are cheap. We test whether the capacity-constrained target recovers the constrained-optimal partition, and whether $\psi_5$ beats cross-entropy for a linear router. The latter mirrors the synthetic top-2 test in which $\psi_5$ reached the Bayes-optimal accuracy of about 0.294 while inconsistent hinges stalled near 0.24 to 0.25.
Pre-training. Decoder models with 64 experts and top-2 routing, at roughly 100M to 1B active parameters, trained on FineWeb. We report validation perplexity, maximum load violation, the fraction of experts receiving under 10% of uniform load, routing regret, and pairwise similarity of expert representations as a homogenization measure.
Fine-tuning. OLMoE-1B-7B on GSM8K and MMLU against ProbMoE and $\phi$-balancing.
Decisive checks. Three results decide the proposal. The first is the diagnostic gate: cold experts must have non-trivial counterfactual competence. The second is that full information plus the consistent target must beat full information plus exploration alone. The third is that the gain must survive FLOP matching. Passing all three would show that expert collapse is partly a mis-specified router objective, fixable with tools from calibration theory. Failing the second would still yield a clean, publishable negative result: exploration, not consistency, is what collapsed routers lack.
References
- Scaling Vision with Sparse Mixture of Experts
- Consistent Estimators for Learning to Defer to an Expert
- Why Ask One When You Can Ask $k$? Learning-to-Defer to the Top-$k$ Experts
- $\phi$-Balancing for Mixture-of-Experts Training
- On the consistency of top-k surrogate losses
- Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer
- When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer
- Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
- BASE Layers: Simplifying Training of Large, Sparse Models
- ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
- Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
- Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers
- Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts
- Advancing Expert Specialization for Better MoE
- Routers in Vision Mixture of Experts: An Empirical Study