Sparse mixture-of-experts language models send each token to one of many feed-forward "experts," with a small learned scorer, the router, choosing which. Left alone, routers collapse: a few experts get most tokens and the rest go unused. Two of the cleanest fixes avoid this by solving a balanced assignment over a whole batch during training. BASE layers solve an exact linear assignment, and Expert Choice routing lets each expert pick its top tokens. Both have the same weakness. When text is generated one token at a time, there is no batch to balance over, so both fall back to a plain per-token argmax over router scores. That decode rule is a different decision rule from the one the model was trained under. We argue it is a statistically inconsistent rule: even with infinite data and a perfect router, it does not recover the balanced assignment the model learned to depend on. The result is expert collapse that appears only at serving time.
The fix follows from linear-programming duality. The population-level balanced assignment is itself a per-token rule. Route each token to $\arg\max_e \big(s(x,e) - \lambda_e\big)$, where $s(x,e)$ is the router score of token $x$ for expert $e$ and $\lambda_e$ is the dual "price" of expert $e$'s capacity constraint. The auction algorithm that BASE already runs computes these prices as a byproduct. Expert Choice's per-expert top-$k$ thresholds play the same role. We propose to carry these prices across batches, use them as the decode rule, and show both theoretically and at small pretraining scale that this closes the gap between training-time and decode-time routing that greedy routing leaves open.
Core Claim
Treat the training-time routing objective as the target loss. For a capacity vector $c$ with $\sum_e c_e = 1$, the target is
$$\pi^\star = \arg\max_{\pi} \; \mathbb{E}_{x}\big[s(x,\pi(x))\big] \quad \text{s.t.} \quad \Pr_x[\pi(x)=e] = c_e \;\; \text{for all } e.$$
Here $\pi$ maps each token representation $x$ to an expert. This is the population limit of what BASE's auction solves per batch, and of what Expert Choice solves with capacity factor 1. Suppose the score differences have no atoms, so ties have probability zero. Then semi-discrete transport duality gives $\pi^\star(x) = \arg\max_e \big(s(x,e) - \lambda^\star_e\big)$ for a unique-up-to-shift price vector $\lambda^\star$.
Two consequences follow. First, the plain greedy decode rule $\arg\max_e s(x,e)$ is consistent with $\pi^\star$ only when $\lambda^\star$ is constant, that is, only when the unconstrained argmax is already balanced. That is exactly the case balancing was introduced to escape. Second, the price-shifted rule with estimated prices $\hat\lambda$ is consistent. Its load deviation and assignment regret shrink as the price estimate converges. Because $\hat\lambda$ comes from an average over the argmax regions of $E$ affine-in-score cells, we expect an empirical-process rate of roughly $\sqrt{E \log(NT)/(NT)}$ over $N$ batches of $T$ tokens.
In the language of An Embedding Framework for Consistent Polyhedral Surrogates, which shows that consistency of a polyhedral surrogate hinges on choosing the right link from continuous scores to discrete decisions, the prices are the correct link. Greedy argmax is the wrong link applied to correct scores.
Relation to Prior Work
The closest work is Mastering Multiple-Expert Routing. It derives surrogates with realizable $H$-consistency (consistency guarantees that hold for the restricted model class actually trained, not only for all functions) for deferring whole inputs to external experts. It tests only on CIFAR/SVHN-scale vision and synthetic data. It has no capacity constraints and no train/decode asymmetry. We borrow its stance that consistency should be checked for the actual decision rule in use, and apply it to internal, capacity-constrained routing.
Efficient and Consistent Adversarial Bipartite Matching shows that matching becomes consistent and tractable when it works with marginals on the Birkhoff polytope rather than with a max-margin surrogate. This motivates working with marginal routing distributions. We do not need its ADMM machinery, because with $E$ experts and many tokens the problem is semi-discrete and its dual has only $E$ variables. The broader inconsistency results for max-type surrogates, On the Consistency of Max-Margin Losses and Consistent Structured Prediction with Max-Min Margin Markov Networks, constrain the design. They warn against fixing the router with a margin-style loss and point us toward fixing the link instead.
On the MoE side, BASE Layers and Expert Choice Routing supply the training-time assignments we analyze. Both document, or are documented to have, the decode fallback and small-batch fragility. $\phi$-Balancing is the most important neighbor. It recasts balancing as a population objective solved by mirror descent with an exponential moving average (EMA) of loads, and its auxiliary loss uses $\nabla\phi(\text{EMA})$, which is effectively a price vector. However, it puts that price into the gradient of a token-choice router rather than into the decision rule. Our contribution is the observation, and the test, that the same population-price idea fixes the decision rule of assignment-trained routers at decode time. That lets those routers keep their throughput advantage without a train/serve mismatch.
$\phi$-Balancing also argues that per-batch balance blocks domain specialization. Advancing Expert Specialization for Better MoE and the orthogonal-optimizer work show that over-strong balancing homogenizes experts. Together these motivate a second use of population prices: replacing the exact per-batch auction during training as well.
Research Program
The theory component states and proves the consistency and inconsistency results above. It covers the top-1 balanced target, the top-$k$ case (where the price-shifted rule selects the $k$ largest shifted scores), and capacity factors above 1, where the constraint becomes an inequality and prices are non-negative. It then derives a finite-sample bound on decode-time load imbalance and on assignment regret relative to $\pi^\star$ when prices are EMA-estimated. We will state honestly where the analysis is weak. The router scores move during training, so the guarantee is for a frozen router at decode time, with a tracking argument for slowly moving scores.
The method component has three variants. The first is price-shifted decoding: keep the trained BASE or Expert Choice model, read auction prices or per-expert top-$k$ thresholds each batch, maintain an EMA, and decode with $\arg\max_e (s - \hat\lambda)$. This costs $E$ subtractions per token. The second is price-shifted training: route training tokens by the same rule with EMA prices instead of the exact per-batch auction. Batch loads are then balanced only in expectation, which lets domain-skewed batches specialize. Whenever a batch exceeds the hardware capacity factor, we use the auction as a fallback. The third is a router-calibration ablation. It adds a small term that trains $s$ so that the price-shifted rule's choices match the batch auction's choices, a direct surrogate for agreement with $\pi^\star$.
Baselines
The naive reference point is BASE and Expert Choice models decoded with plain greedy argmax, as their papers prescribe. Running the same checkpoints in "oracle" mode, with the full-batch auction applied to teacher-forced evaluation text, gives an upper reference. The gap between the two is the quantity we claim to close. No reported result found on Lacuna for the perplexity or load gap between batch-assignment and greedy routing in either paper, so measuring it is itself a first deliverable.
The strongest existing competitor is $\phi$-balancing with a token-choice router $\phi$-Balancing. It never has a decode mismatch and already uses population prices, though in the loss. Its best reported numbers in the file are fine-tuning results on Moonlight-16B-A3B: Math500 accuracy from 64.8% to 66.2% and HumanEval from 73.2% to 75.9% over Switch-style losses, with wins in over 80% of 21 setups. We cannot fine-tune at that scale, so the fair comparison is pretraining at its smallest reported Gemma-style scale (around 111M active parameters) on C4. All methods use identical data order, token budget, expert count, capacity factor, and an equal hyperparameter sweep budget; for $\phi$-balancing that sweep covers the coefficient and potential choice. Switch-style auxiliary loss is included as the industry default. For throughput, BASE reports 545k tokens/s against 469k for Switch Transformer BASE Layers.
Beating the field means one of two outcomes. Either price-shifted BASE matches $\phi$-balancing's decode perplexity within noise while keeping BASE's training throughput, or it beats $\phi$-balancing's perplexity at matched compute. If it does neither, assignment-trained routing has no reason to exist once $\phi$-balancing is available, and we will report that.
The key-ingredient ablation replaces dual prices with a per-expert bias learned by gradient or by a simple load-feedback rule, applied at decode only. If such a bias matches prices, the contribution reduces to "add a bias," and the theory merely explains why biases work. If dual prices are better, the link derived from the target matters.
Why not just…?
Why not just train with token-choice $\phi$-balancing and avoid the mismatch entirely? This is the real threat. If the throughput and specialization advantages of exact assignment are illusory at matched compute, our decode fix rescues a method nobody needs. The matched pretraining comparison above settles it directly. We consider the proposal successful only if price-shifted BASE or Expert Choice is at least on par in decode perplexity and better in throughput or downstream accuracy.
Why not buffer tokens and run the auction at inference? Prefill can do this, but streaming decode cannot without adding latency, and the small-batch weakness noted for Expert Choice Expert Choice Routing means a short buffer gives noisy assignments anyway. We include a buffered-auction decode baseline at buffer sizes from 8 to 512 tokens to show where it crosses the price-shifted rule.
Why not fine-tune the router with an auxiliary loss after pretraining so greedy routing becomes balanced? That changes which expert each token reaches, and the downstream experts were trained on the assignment-routed distribution. A post-hoc auxiliary loss therefore moves tokens away from the experts trained for them, whereas the price rule reproduces $\pi^\star$ itself. We test it as a baseline with an equal fine-tuning token budget, measuring perplexity and agreement with the oracle assignment.
Why bother with consistency theory if it is asymptotic? Following the move to model-class-specific guarantees in Structured Prediction with Stronger Consistency Guarantees, our bound is finite-sample in the number of tokens used to estimate prices. A synthetic check tests whether the bound's predicted rate holds empirically.
Proposed Experiments
The first study is synthetic. Use a mixture-of-Gaussians token distribution with fixed linear experts and known scores, in the spirit of the realizable benchmark in Mastering Multiple-Expert Routing, and compute $\pi^\star$ exactly. Then plot assignment regret and load deviation against price-estimation tokens for greedy, learned-bias, and price-shifted rules. The decisive prediction is that greedy regret plateaus at a nonzero level while price-shifted regret decays at roughly the predicted square-root rate.
The main study pretrains MoE models on C4, with FineWeb-Edu The FineWeb Datasets as a second corpus, at roughly 100M and 400M active parameters with 16 to 64 experts. It covers BASE, Expert Choice, Switch auxiliary loss, $\phi$-balancing, and our two price-shifted variants.
The metrics are as follows. Validation perplexity is measured under true autoregressive decode routing. Decode-time load imbalance is reported as the Gini coefficient, the metric used in $\phi$-balancing, and as the maximum-to-mean expert load. Agreement with oracle batch assignment is also measured. Training throughput is reported in tokens per second. Domain specialization is measured as the mutual information between expert and source domain (code versus prose versus academic text). Downstream evaluation at this scale uses ARC and MMLU as in FineWeb, plus GSM8K and HumanEval if the 400M models are non-trivial on them.
The decisive checks are three. First, greedy decoding of assignment-trained models must show measurable decode-time collapse, meaning a higher Gini and a perplexity gap relative to oracle; without this there is no problem to solve. Second, price-shifted decoding must close most of that gap and beat the learned-bias ablation. Third, price-shifted training must yield higher domain specialization than the exact per-batch auction at equal perplexity or better. If the first check fails, we will report that assignment routers are more robust to the mismatch than assumed and stop there.
References
- An Embedding Framework for Consistent Polyhedral Surrogates
- Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer
- Efficient and Consistent Adversarial Bipartite Matching
- On the Consistency of Max-Margin Losses
- Consistent Structured Prediction with Max-Min Margin Markov Networks
- BASE Layers: Simplifying Training of Large, Sparse Models
- Mixture-of-Experts with Expert Choice Routing
- $\phi$-Balancing for Mixture-of-Experts Training
- Advancing Expert Specialization for Better MoE
- Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
- Structured Prediction with Stronger Consistency Guarantees
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale