Trust-Region Routers: Levenberg-Marquardt Damping for Stable, Communication-Aware Mixture-of-Experts Gating

A mixture-of-experts model is a neural network that sends each token to a few of many "expert" sub-networks. The small model that makes this choice is the router. When experts live on different GPUs, every layer needs an all-to-all exchange, in which each device ships its tokens to the devices that hold the chosen experts and then collects the results. This exchange often takes more than half of training time. The router causes much of that cost, in two ways. It keeps changing its mind: StableMoE reports that about 30% of tokens are still switching experts midway through training. It also settles into habits that are expensive to undo, such as sending tokens over slow links between machines. Current fixes either freeze the router early (StableMoE, Grouter) or perturb it at random (Gating Dropout). Freezing buys stability and predictable traffic, and it gives up the ability to adapt later in training.

We propose an optimizer used only for the router. It treats each router update as a trust-region step: a step that is trusted only as far as a local model of the loss has proven accurate. The step is damped in the Levenberg-Marquardt style, so it moves smoothly between a cautious gradient step and a confident curvature-corrected step. The novel part is what sets the damping. Besides the usual check of predicted against actual loss reduction, the damping also limits how many tokens a step may reassign and how much cross-node traffic the result creates. Link congestion enters through prices on each network link, and these prices are updated multiplicatively. In this design the router freezes itself when moves do not pay off and loosens when they do. That makes freezing an adaptive outcome of the optimizer rather than a fixed schedule.

Core Claim

Routing fluctuation, router lock-in, and all-to-all congestion are three symptoms of one problem. The router takes steps that its local loss model cannot justify. Our claim is that a damped Gauss-Newton router update can match the throughput benefit of frozen routing while keeping the quality benefit of adaptive routing. The update is governed by three things: a gain ratio, a cap on token reassignment, and link prices updated with multiplicative weights. The claim is falsified if the controller's damping simply rises to "frozen" and stays there, or if a well-tuned first-order router with the same churn cap does just as well.

Mechanism

Let $\theta$ denote one layer's router parameters, $g$ the stochastic gradient of the training loss with respect to $\theta$, and $G$ a Gauss-Newton curvature estimate for the router alone. For a linear router producing logits $z = Wx$ over $E$ experts, $G$ factors as a Kronecker product. One factor is the input second-moment matrix, of size $d_{model}\times d_{model}$. The other is the softmax curvature $\mathrm{diag}(p)-pp^\top$ over experts, of size $E\times E$. Both are small, so the curvature costs little even though curvature for the whole network would be out of reach. The router step is

$$\Delta\theta = -(G + \mu D)^{-1} g,$$

where $D$ is the diagonal of $G$ and $\mu$ is the damping. The diagonal damping follows the maximum-diagonal variant in Improving Levenberg-Marquardt Algorithm for Neural Networks. After a step we compute two quantities. The first is a lagged gain ratio $\rho$: the realized loss decrease divided by the decrease the quadratic model predicted, smoothed over a few steps because a stochastic loss is noisy. The second is the churn $\kappa$: the fraction of tokens in a small cached probe batch whose top-$k$ expert set changed. The damping rises multiplicatively when $\rho$ is low or $\kappa$ exceeds a budget $\kappa_{max}$, and falls when the step paid off within budget.

Congestion enters as nonnegative prices $\lambda_\ell$ on each link $\ell$, which are subtracted from the logits in proportion to the bytes a routing choice would move over that link. The prices are updated by

$$\lambda_\ell \leftarrow \lambda_\ell \exp\big(\eta\,(T_\ell / C_\ell - 1)\big),$$

where $T_\ell$ is the measured traffic on the link, $C_\ell$ is its capacity budget, and $\eta$ is a step size. This is the multiplicative-weights rule for approximately satisfying packing constraints, and its log-regret behavior is why we use it here.

We are deliberate about where multiplicative updates belong in this design. Applying multiplicative updates to routing probabilities is the same as making additive updates to logits. The attribute-efficiency results (Worst-case Loss Bounds for Single Neurons, The limits of squared Euclidean distance regularization) therefore give no special advantage there. Their real advantage, fast suppression of the few active constraints among many, applies to the link prices, where most links are idle and a handful are congested.

Local-minimum entrapment is handled by the uphill-step acceptance of Improving Levenberg-Marquardt Algorithm for Neural Networks. A step with $\rho<0$ is accepted if it agrees with the router's momentum and stays within the churn budget. The router can then climb out of a costly co-adapted assignment without the blind randomness of gating dropout. Finally, when churn stays below budget for a window of $m$ steps, a planner re-places experts offline, as Grouter does once. It then reuses that placement until churn rises again. This borrows the "stale but reused" snapshot idea of Gram-reduced Levenberg-Marquardt, which recomputes its curvature decomposition only every $m$ iterations. That paper explicitly lists the distributed, latency-dominated setting as open, and this is the setting we address.

Research Program

We will build the router optimizer as a drop-in replacement in a standard expert-parallel training stack. All non-router parameters stay on AdamW. The work has four analysis questions.

The first is whether the lagged, stochastic gain ratio is informative at all. Stochastic Levenberg-Marquardt is the gap that Gram-reduced Levenberg-Marquardt leaves open. If $\rho$ turns out to be pure noise, the controller reduces to a churn cap.

The second is how the damping behaves over training. We expect high damping early, when representations shift and steps fail; a phase of low damping in which the router specializes; and late periodic loosening. The late loosening is the regime that frozen routers give up.

The third is whether multiplicative link prices beat the fixed soft penalty of TA-MoE in how quickly they react to congestion.

The fourth is whether perturbed cosine routing interacts with the trust region. Perturbing Cosine Router shows that adding a small perturbation to the normalization breaks a parameter-tangling identity that makes the loss blind to some router changes. That same identity should make $G$ near-singular along those directions, and damping is a second way to handle that degeneracy.

Baselines

The naive reference point is a standard top-$k$ linear router trained with AdamW and a load-balancing auxiliary loss, in the GShard style (GShard). We also report the auxiliary-loss-free balancing variant that Grouter compares against.

The strongest competitor on communication is TA-MoE, which adds a topology-aware soft penalty to the router objective. It is the most direct existing answer to "make the router avoid slow links." On OpenWebText2 with a GPT-3 Medium MoE, it reports communication cost "reduced by up to 6.4x," convergence to target validation loss "1.25x to 1.54x faster" than FasterMoE, and perplexity that "remained consistent with baseline models." Beating it means matching or improving its wall-clock time to target validation loss and its cross-node bytes, at no worse final perplexity, on the same cluster topology.

The strongest competitor on stability is StableMoE, which freezes a distilled router after 10% of training. It reports test perplexity "18.93" against "19.25" for Hash Layer on its 100B-token corpus, and average BLEU "30.32" on WMT-10.

On communication skipping, we compare against Gating Dropout, which reaches baseline best BLEU "58.48% faster" on WMT-10.

Grouter is a reference rather than a like-for-like competitor, because it needs a converged teacher. It reports matching baseline validation loss with "only 23.3% of the data" and throughput gains of "15.5% to 33.5%" on C4. We will run it where a teacher of matching architecture exists and label it as teacher-assisted.

Three ablations isolate our ingredients. The first keeps the curvature preconditioner and fixes the damping, which is K-FAC-style router preconditioning. The second keeps the churn and gain-ratio controller and drops the curvature, giving an Adam step whose length is scaled to satisfy the same rule. The third replaces multiplicative link prices with TA-MoE's fixed penalty. Every method gets the same data, token budget, cluster, and tuning budget: an equal number of trials over learning rate and that method's own key hyperparameter, whether $p$, penalty weight, freeze point, or $\kappa_{max}$.

Why not just…?

Why not just freeze the router early, as StableMoE does? Freezing is the method to beat on throughput, and on short runs it may well win. Our bet is that the advantage of adaptivity appears late in training and in domain shift, which Perturbing Cosine Router and ProbMoE argue frozen or collapsed routing harms. We test this by continuing pretraining on a shifted domain after the main run. If the trust-region router shows no gain there over a frozen router, the method is not worth its complexity.

Why not just add TA-MoE's penalty and keep AdamW? The fixed penalty weight is a compromise across training. When it is too weak, congestion persists. When it is too strong, quality suffers. Nothing in it limits churn. The prices-only ablation (multiplicative prices with an Adam router) separates the price mechanism from the trust region. If that ablation alone matches the full method, the curvature and damping are unnecessary and we will say so.

Why not just cap churn with a plain first-order step-size rule? This is the most serious threat. It may be that the reassignment cap does all the work. The second ablation above tests exactly this. The full method must beat it on time to target loss, or on the fluctuation measure of StableMoE, to justify the Gauss-Newton curvature.

Why not just randomize, as Gating Dropout does? Random skipping explores without regard to whether a move helps. It also gives nothing at inference, because it is switched off. Our uphill acceptance is directed exploration, and the trained router keeps its locality at inference. We report inference-time cross-node bytes for both methods.

Proposed Experiments

The primary testbed reproduces TA-MoE's OpenWebText2 GPT-3 Medium MoE setup on a two-tier cluster with fast links inside machines and slower links between them. We measure wall-clock time to TA-MoE's target validation loss, cross-node bytes per step, final perplexity, routing churn over training, and the trajectory of the damping $\mu$.

The second testbed is WMT-10 translation, following StableMoE and Gating Dropout. We measure BLEU against StableMoE's 30.32 and wall-clock time to Gating Dropout's best-BLEU point.

A third, smaller check runs the continued-pretraining domain shift on C4 followed by code or math text. Its purpose is to probe the adaptivity claim, with GSM8K-style evaluation as a downstream signal. For GSM8K, the only number we have is ProbMoE's fine-tuning result of "50.19%" against "45.94%" on OLMoE-1B-7B. That setting is not comparable to ours, so we cite it only as evidence that routing quality moves reasoning accuracy.

Three outcomes would be decisive. The first is that the full method beats both TA-MoE and the churn-capped Adam ablation on time to target loss. The second is that it matches or beats StableMoE's perplexity while its damping visibly loosens late in training. The third is that its advantage over frozen routing grows under domain shift. If $\mu$ simply saturates high and stays there, our method has rediscovered freezing at extra cost, and we will report that negative result.