Link-Priced Expert Balancing for Mixture-of-Experts Training over Constrained Interconnects

A mixture-of-experts (MoE) model replaces one large feed-forward block with many smaller "experts," plus a small learned router that sends each token to one or two of them. When experts sit on different machines, every routing decision is also a network transfer. Two failures are well documented. In the first, the router collapses onto a few favored experts, which starves the rest and overloads the machines that host the favorites. In the second, the balancing penalties meant to prevent that collapse push all experts toward the same function.

The best current fix, population-level balancing with a convex potential, charges each expert a "congestion price" computed from its long-run usage. It assumes every expert is equally cheap to reach. Real clusters, and even more so edge or geo-distributed deployments, connect machines through links whose capacities differ, change over time, and are often far below intra-node bandwidth. This is the setting the backhaul literature studies for radio networks.

We propose to move the congestion price from experts to links. The router pays a price for each network link a token would cross, estimated from long-run traffic relative to that link's current capacity. It also keeps an entropy-based price on each expert's overall usage, so that no expert behind a slow link is allowed to die. The weight between "route well" and "route cheaply" is not a tuned constant. It is set in closed form by the quadratic transform from fractional programming, applied to the ratio of routing quality to step time. The testable claim is that when token sources are spread across nodes, this link-level formulation reaches a given validation loss in less wall-clock time than expert-level balancing re-weighted by capacity. It should do so without the "locality collapse" that naive network-aware routing causes.

Core Claim

$\phi$-Balancing for Mixture-of-Experts Training supplies the method we build on. It replaces noisy per-batch balancing losses with a strictly convex potential over population routing frequencies. Negative entropy works best. The potential's gradient is evaluated at an exponential moving average (EMA) of expert usage, and the resulting per-expert prices are updated by online mirror descent. The paper explicitly assumes homogeneous capacity and unconstrained communication.

Our claim is that per-expert prices cannot express what matters under constrained interconnects. In standard expert-parallel training, tokens originate on every node. The cost of routing a token to expert $e$ therefore depends on the pair (source node, host node of $e$), not on $e$ alone. Two tokens choosing the same expert can impose very different network loads.

Pricing links rather than experts creates a new failure mode. If remote links are expensive, each node's tokens drift toward its own local experts. Experts then specialize to the node's data shard, and remote experts become cold. We call this locality collapse. It is a network-induced version of expert collapse, and neither the balancing literature nor the network-aware inference literature measures it. The proposal is to control both effects with one potential. Let $u_e$ be the population fraction of tokens sent to expert $e$. Let $f_{sn}$ be the population fraction of all tokens that travel from source node $s$ to host node $n$, and let $C_{sn}(t)$ be that link's capacity at time $t$, in the same token-rate units. The potential is

$$ \Phi_t(u, f) = \sum_{e} u_e \log u_e \;+\; \sum_{s \neq n} \psi\!\left(\frac{f_{sn}}{C_{sn}(t)}\right), $$

where $\psi$ is a convex barrier on link utilization, for example $-\log(1-\rho)$ below saturation. Following the $\phi$-balancing recipe, the per-batch penalty is the router's batch probabilities dotted with $\nabla \Phi_t$ evaluated at the EMA estimates $(\bar u, \bar f)$. This gives each token a routing cost equal to its expert's usage price plus the price of the link it would cross.

Research Program

The first component is the link-priced balancer itself. It needs per-link EMA traffic counters, which are cheap because they are already implied by the all-to-all dispatch counts. It also needs a capacity estimate $C_{sn}(t)$, measured passively from dispatch timings or supplied by a trace.

The time variation matters. The backhaul literature shows that static partitioning fails when link capacity fluctuates (Cell-Free Massive MIMO with Limited Backhaul; Uplink Multi-Cell Processing: Approximate Sum Capacity under a Sum Backhaul Constraint). Those papers optimize quantization and receiver design for radio signals, not learned routing. We borrow only their lesson that transport limits must be priced dynamically.

The second component sets the trade-off weight. What a practitioner cares about is learning progress per second. We model this as the ratio of expected routing quality $A$ to a smooth step-time model $B$, a log-sum-exp over link loads divided by capacities, which approximates the straggler maximum. Quadratic Transform for Fractional Programming in Signal Processing and Machine Learning shows that maximizing such a ratio is exactly a majorization-minimization scheme. It replaces $A/B$ with $2y\sqrt{A} - y^2 B$, with closed-form $y = \sqrt{A}/B$. That paper applies the transform to power control and beamforming. We apply it to the scalar that multiplies the link-price term: the congestion weight becomes $y^2$, recomputed from EMA quantities every few hundred steps. The weight rises when links saturate relative to the quality being gained and falls when they do not.

Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts is a constraint on this design. It shows that softmax-gated MoE losses lack the smoothness classical MM proofs need. We therefore do not claim convergence of the full network. Monotonicity is claimed only for the scalar ratio subproblem, where the quadratic transform's guarantee genuinely applies.

The third component is analysis. We will characterize when link pricing reduces to expert pricing. In a star topology, where all tokens start at one hub and each expert sits behind its own link, the two are equivalent, and a capacity-weighted expert target should suffice. We will also measure locality collapse with two quantities: the mutual information between a token's source node and its chosen expert, and the fraction of experts whose EMA usage falls below a small threshold.

Baselines

The naive reference point is standard token-choice routing with a Switch-style auxiliary loss, dispatched with flat all-to-all. This is what most runs do by default. HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System reports that flat all-to-all can consume most of training time on commodity clusters. Its hierarchical all-to-all gave a 2× speedup at 64 GPUs. We therefore run every method, including this reference, on hierarchical all-to-all as well, so that routing gains are measured on top of the obvious systems fix.

The strongest existing method is $\phi$-balancing with negative entropy ($\phi$-Balancing for Mixture-of-Experts Training). In fine-tuning Moonlight-16B-A3B, it raised Math500 accuracy from 64.8% to 66.2% and HumanEval pass@1 from 73.2% to 75.9% over auxiliary-loss balancing. It won in over 80% of 21 fine-tuning setups. For its pre-training runs on C4 with 111M–986M active parameters, no reported loss or perplexity figure was found on Lacuna. We will therefore reproduce it ourselves at matched model size, data, token budget, and a matched hyperparameter sweep. We run it in two forms: as published, and with its uniform target replaced by a capacity-proportional target. The second form is the fairest version of "just make $\phi$-balancing capacity-aware." Beating it means reaching the same validation loss in at least 15% less wall-clock time under heterogeneous links, with no worse final loss at a fixed token budget and no higher dead-expert fraction.

The hard-constraint competitor is BASE-style assignment (BASE Layers: Simplifying Training of Large, Sparse Models). That paper solves an exact balanced assignment per batch with an auction algorithm and reported 545k tokens/s against 469k for Switch on 128 GPUs. We extend it to per-link capacity caps, which turns the per-batch problem into a small transportation problem solved with Sinkhorn iterations. It is the natural "just enforce the constraint" alternative. Its known weakness is that inference must fall back to greedy routing, so we evaluate it with both training-time and greedy inference routing.

For network-aware selection, we include a training-time adaptation of the Weight-to-Latency Ratio from WDMoE: Wireless Distributed Mixture of Experts for Large Language Models. That paper divides router weight by estimated link and compute latency for edge inference and reports a 45.75% latency reduction on PIQA. It is the most direct existing heuristic for routing by link cost. We expect it to exhibit locality collapse, since it has no usage-level anti-collapse term.

Two ablations isolate our ingredients. The first removes the entropy term, keeping link prices only, to test whether the expert-marginal potential is what prevents locality collapse. The second replaces the quadratic-transform weight with the best fixed weight found by the same sweep budget, to test whether adaptive weighting matters under time-varying capacity.

Why not just…?

Why not just give $\phi$-balancing a capacity-weighted target distribution? This is the most serious threat, and in the star topology we expect it to be equivalent to our method. The distinction appears only when tokens originate on many nodes. There, a single expert-level price charges a local token and a cross-link token the same amount, even though only one of them uses the slow link. The experiment that settles it runs both methods on a star topology and on a multi-node mesh with the same total bandwidth. If the capacity-weighted variant matches us on the mesh too, the link-level formulation is unnecessary, and we would report that as the finding.

Why not just fix it at the systems layer with hierarchical all-to-all and capacity-aware token dropping? Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts drops the lowest-scored overflow tokens, or reroutes them to experts on the same device. It reports a 1.85× speedup on Mixtral-8×7B-Instruct at slightly improved accuracy. These fixes act after the router has decided. They do not change what the router learns, and inside training, dropped tokens are lost gradient signal. We run all methods on hierarchical all-to-all, and we also combine token dropping with each routing method. If our gain vanishes once dropping is added, the routing-level contribution is not worth its complexity.

Why not just enforce per-link capacity exactly each batch, BASE-style? Exact per-batch constraints inherit the noise problem that $\phi$-balancing identified: a mini-batch is an unrepresentative sample, so hard balancing on it distorts specialization. They also create a train–inference mismatch, because the assignment needs the whole batch. The link-capped Sinkhorn baseline tests this directly on specialization metrics and on the gap between training-time and greedy-inference quality.

Proposed Experiments

We pre-train decoder-only MoE models at two scales, roughly 150M and 1B active parameters with 64 experts, matching the range $\phi$-balancing used. Training data is FineWeb (sample-10BT and sample-100BT splits; The FineWeb Datasets). Runs use 4 nodes of 8 GPUs, with inter-node bandwidth shaped in software into three regimes. In the homogeneous regime all links are fast, as a sanity check where all methods should tie. In the static heterogeneous regime one node pair sits at one-eighth bandwidth. In the time-varying regime capacities follow fading-style traces, including an abrupt halving of one link mid-training.

The primary metric is validation loss against wall-clock time. Secondary metrics are p50 and p99 step time, inter-node bytes per token, and expert-usage entropy. We also track the dead-expert fraction and two specialization measures: node–expert mutual information for locality collapse, and domain–expert mutual information for genuine specialization. Downstream zero-shot evaluation uses GSM8K, MMLU, ARC-Challenge, and HellaSwag at the 1B scale. If compute allows, a fine-tuning track on Moonlight-16B-A3B with Math500 and HumanEval lets us compare against the 66.2% and 75.9% that $\phi$-balancing reports.

Four outcomes are decisive. First, the WLR-style and link-only ablations should show rising node–expert mutual information and dead experts behind the slow link, while the full method should not. Without this, the locality-collapse story is wrong. Second, the full method should beat capacity-weighted $\phi$-balancing on the mesh but not on the star; any other pattern falsifies the core claim. Third, after the mid-training capacity drop, the full method should restore step time within a few thousand steps without killing the experts behind the degraded link. Fourth, because the learned prices are population-level constants, the trained router should need no special handling at inference. We check this by showing that greedy decoding quality matches training-time routing quality, which the link-capped BASE baseline is expected to miss.