A mixture-of-experts (MoE) model replaces one large feedforward block with many smaller "experts" and a small gating network that sends each token to a few of them. Two things commonly go wrong. In expert collapse, a few experts end up handling most tokens while others get almost none and stop learning. To keep the hardware fast, each expert is also given a fixed number of slots per batch, its capacity, and tokens beyond that number are dropped. Current balancing methods equalize the average number of tokens each expert receives. Dropping depends on something else: how often an expert's load spikes past its capacity. Two experts with the same average load can lose very different fractions of their tokens if one sees bursty traffic, for example an expert that specializes in code and is flooded whenever a batch contains a code file.
Wireless communication has a precise name for this situation. A transmitter that cannot see the current state of the channel picks a rate blindly, and the link suffers an "outage" when the realized channel is worse than planned. Each token's routing decision is made the same way. The gating network scores the token without knowing how many other tokens in the batch will pick the same expert. In distributed federated learning over fading links, Quantized Federated Learning under Transmission Delay and Outage Constraints showed that clients with higher outage probability are silently down-weighted in the aggregated gradient, and that equalizing outage probability across clients removes this bias. We propose to carry that principle into MoE training. If specialist experts are bursty, they will be dropped more often, receive weaker gradients, and drift toward starvation or toward generic behavior. Balancing should therefore target equal drop probability per expert rather than equal mean load.
Core Claim
Let $L_i$ be the random number of tokens that select expert $i$ in a batch, $c_i$ its capacity, and $p_i = \Pr(L_i > c_i)$ its drop probability, which we call its outage. Under token-choice routing with capacity limits, the expected gradient reaching expert $i$ is scaled by roughly $(1 - p_i)$ relative to its routed demand. When $p_i$ varies across experts, the effective training objective therefore reweights data domains, and it does so against the bursty, specialized ones. Existing balancers only control the mean, $\mathbb{E}[L_i]$. This includes the loss-free bias method in Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, which nudges per-expert routing biases from last-batch load, and the population-level potential game in $\phi$-Balancing for Mixture-of-Experts Training, which tracks an exponential moving average of expected load. Neither controls the tail $p_i$.
Our claim has two parts. The first is that balanced-mean MoE training leaves substantial and persistent dispersion in $p_i$, correlated with expert specialization. The second is that allocating capacity to equalize outage removes this hidden reweighting and reduces both starvation and homogenization at matched compute. The allocation solves
$$ \min_{c_1,\dots,c_E} \ \max_i \ \Pr(L_i > c_i) \quad \text{subject to} \quad \sum_{i=1}^{E} c_i = C, $$
where $E$ is the number of experts and $C$ is the total slot budget fixed by the capacity factor.
This differs from the closest existing work in two ways. The federated-learning paper solves a related allocation over bandwidth and quantization bits for dense models with whole-vector uploads, and never touches routing. Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts drops tokens deliberately at serving time to avoid slow devices, but uses a single shared capacity factor $\gamma$ and does not change training. Our proposal changes how capacity is split across experts during training, using a load-tail signal that no current balancer uses.
Research Program
The work has three stages. The third stage depends on the first.
The first stage is a measurement study. We train small token-choice MoE language models on FineWeb-Edu with top-2 routing, 16 to 64 experts, and capacity factors between 1.0 and 1.5. We use three balancers: the Switch-style auxiliary loss, the loss-free bias method, and $\phi$-balancing. Throughout training we log per-expert mean load, per-expert drop rate $p_i$, the coefficient of variation of load across batches, and a specialization score. The specialization score is the entropy of an expert's routing distribution over FineWeb-Edu domain or source labels and over a held-out code and math mix. The hypothesis is falsified if, under mean-balanced training, the spread of $p_i$ is negligible (say, maximum over minimum below 1.5) or uncorrelated with specialization. In that case we stop and report the negative result.
The second stage is the method. For each expert we maintain an exponential moving average of a high quantile of $L_i$, which we estimate cheaply from per-batch counts. Per-expert capacities $c_i$ are then set so that the estimated $p_i$ are equal, with the sum fixed at $C$. This amounts to water-filling on the load distributions, the same structure as power allocation under outage constraints. Total slots, and therefore FLOPs, match the uniform-capacity baseline. The mean-load balancer (loss-free bias) stays in place, so the only change is how a fixed budget of slots is divided.
We also test a second correction that removes the bias without reallocating capacity. Kept tokens of expert $i$ have their expert-output gradient scaled by $1/(1-\hat p_i)$, which is inverse-propensity reweighting. This is the cheap alternative a reviewer will propose, and it is a real ablation.
For heterogeneous capacity in expert-parallel training, per-expert $c_i$ changes the per-device load. We report the device-level maximum load as a straggler metric and constrain the allocation so that per-device slot sums stay balanced. This extends naturally toward the heterogeneous-capacity setting that the $\phi$-balancing authors leave open.
The third stage is a robustness study under unknown serving capacity, which is the "no channel knowledge" regime. At deployment the effective capacity often differs from training, so we evaluate every trained model under a sweep of inference capacity factors using the drop rule from Capacity-Aware Inference. We also include a variant trained with capacity factor sampled randomly per step, so that the model never knows its capacity in advance. Here Minimum Expected Distortion in Gaussian Layered Broadcast Coding with Successive Refinement supplies a prediction. That work shows that when channel quality is unknown, extra resources should protect a robust base layer while higher refinement layers keep a fixed allocation. Its MoE counterpart is the always-active shared expert studied in DeepSeekMoE and in Decoding Knowledge Attribution in Mixture-of-Experts. The prediction is that the optimal width of the shared expert should grow with the variance of the capacity distribution. We test this with a small grid over shared-expert width. This is a secondary test of the analogy and not the main claim.
Baselines
The naive reference is a token-choice MoE with the Switch-style auxiliary balancing loss and uniform capacity, tuned over the loss weight. The upper reference is the same model with capacity factor raised until drops are essentially zero. That reference shows how much of any gap is caused by dropping at all. It is reported at its true, higher compute cost and not counted as a competitor at matched compute.
The strongest existing method is Loss-Free Balancing from Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. It is the most direct competitor because it already removes interference gradients and achieves near-perfect mean balance. On their 1B MoE trained for 200B tokens, they report validation perplexity 9.50 against 9.56 for the auxiliary-loss baseline, and Max Violation 0.04 against 0.72. Our method sits on top of their bias update, so the comparison isolates capacity allocation. Both runs use identical data order, total slots, steps, and a matched tuning budget, with the same number of runs over the bias update rate.
We also compare against $\phi$-balancing, which reports Math500 rising from 64.8% to 66.2% and HumanEval from 73.2% to 75.9% on Moonlight-16B-A3B. Those numbers come from a far larger model than ours, so we reimplement it at our scale rather than compare against them.
Two further baselines change the routing direction or the drop rule. Expert-choice routing, analyzed in Routers in Vision Mixture of Experts: An Empirical Study, equalizes expert load by construction but moves outage onto tokens, some of which receive no expert. We report its token-side drop dispersion across domains for that reason. Batch Prioritized Routing from Scaling Vision with Sparse Mixture of Experts drops low-confidence tokens first rather than arbitrary ones.
The key ablations isolate each ingredient. Inverse-propensity reweighting alone tests whether gradient correction without reallocation suffices. Outage-equalized capacity without the loss-free bias tests whether tail control alone is enough.
Beating Loss-Free Balancing means three things together. Drop-rate dispersion must be lower at matched FLOPs and matched device-level maximum load. Validation perplexity must be lower by more than the run-to-run spread across three seeds. Per-domain perplexity on the bursty domains must improve, since those are the domains the hypothesis says are being down-weighted.
Why not just…?
Why not just raise the capacity factor so nothing is dropped? No drops means no outage bias, which is why this setting is our upper reference. However, it raises activation memory and all-to-all traffic for every expert in order to serve the tail of a few. The straggler analysis in Capacity-Aware Inference shows these costs dominate in expert-parallel systems. The relevant question is what capacity factor each method needs to reach a given perplexity. If uniform capacity at factor 1.25 matches outage-equalized capacity at 1.0, our method saves that margin. If the curves coincide, the idea fails.
Why not just reweight the gradients of dropped experts instead of reallocating capacity? This is the most serious threat, because it is simpler and fixes the expected gradient. Its weakness is variance. A heavily dropped expert's gradient is computed on few tokens and then amplified, and which tokens survive is itself biased toward the start of the batch or toward high scores. We run it as a full ablation. If it matches our method on perplexity and specialization, we will recommend it as the outcome and report the capacity reallocation as unnecessary.
Why not just rely on Loss-Free Balancing or BASE-style exact assignment, which already balance load? Loss-Free Balancing controls mean load, and the measurement stage directly tests whether that leaves tail dispersion behind. Exact assignment in BASE Layers forces $T/E$ tokens per expert, which overrides routing preferences, the pressure toward homogenization documented in Advancing Expert Specialization for Better MoE. The BASE authors also acknowledge that it must revert to greedy top-1 routing at inference, which reintroduces uncontrolled load. If the measurement stage shows that Loss-Free Balancing already yields uniform $p_i$, this proposal has no reason to exist, and we say so up front.
Decisive Checks
The first check comes from the measurement stage. After mean balancing, does drop rate vary across experts, and does it correlate with domain specialization? A null result here ends the project.
The second check is the matched-compute comparison against Loss-Free Balancing. It reports perplexity, per-domain perplexity on code and math versus general web text, the dead-expert count (experts below 1% of uniform share), and pairwise expert weight similarity as a homogenization measure, in the spirit of Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer.
The third check is robustness. We sweep inference capacity factors and block top experts, following the attribution protocol of Decoding Knowledge Attribution in Mixture-of-Experts, which reports reciprocal-rank drops of 43% in Qwen 1.5-MoE and 76% in OLMoE when the top-10 experts are blocked. We expect outage-equalized models to degrade more gracefully, because no expert was trained on a systematically thinned stream.
For downstream accuracy at this scale, the FineWeb paper reports MMLU of 33% for base FineWeb and 37% for FineWeb-Edu. Those are dense-model figures from The FineWeb Datasets, so they serve as a sanity range only. We found no reported result on Lacuna for outage-dispersion metrics in MoE.
References
- Quantized Federated Learning Under Transmission Delay and Outage Constraints
- Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
- $\phi$-Balancing for Mixture-of-Experts Training
- Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts
- Minimum Expected Distortion in Gaussian Layered Broadcast Coding with Successive Refinement
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
- Routers in Vision Mixture of Experts: An Empirical Study
- Scaling Vision with Sparse Mixture of Experts
- BASE Layers: Simplifying Training of Large, Sparse Models
- Advancing Expert Specialization for Better MoE
- Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale