Router-Only Sharpness-Aware Training for Mixture-of-Experts at Near-Zero Cost

Mixture-of-experts (MoE) language models replace each dense feed-forward block with many parallel sub-networks, called experts, plus a small learned gate, the router, that sends each token to only a few of them. A persistent failure in these models is expert collapse. Early chance advantages get amplified until a handful of experts handle most tokens and the rest stop learning. The standard fix is a load-balancing penalty, but it pushes the surviving experts to become near-copies of one another. Separately, sharpness-aware minimization (SAM) trains networks toward flat regions of the loss, where small weight changes barely move the loss. SAM reliably improves generalization, but it doubles training cost, and that cost defeats the purpose of a sparse model.

This note proposes applying SAM's worst-case weight perturbation to the router alone. The perturbed gradient would be computed from quantities the ordinary backward pass already produces, so no second forward or backward pass through the experts is needed. The bet is that collapse is a flatness problem located in the router: allocation runs away because tiny router changes flip where tokens go. Flattening the router with respect to the task loss should then stabilize allocation without forcing uniform load. The extra cost should be a small fraction of a training step, because routers hold a tiny share of the parameters.

Core Claim

Our hypothesis is that the router's loss surface is sharp at the moments collapse begins, and that a cheap, locally computed SAM step on router weights reduces dead experts and expert redundancy better than load-balancing penalties, temperature tuning, or noisy gating. It should do so at under roughly 5% step-time overhead, a target modeled on the 2–3% reported for head-only perturbation. The file contains no work that applies sharpness-aware optimization to MoE routing.

The closest precedent is logits-SAM. It perturbs only the final language-model head during preference tuning and adds about 2–3% overhead. It is cheap only because the head is the last layer, so re-evaluating it after perturbation touches nothing downstream. A router sits inside the network, and naively perturbing it requires re-running every later layer. The technical contribution here is a way around that.

Research Program

The router surrogate. Take an MoE layer with output $y(x)=\sum_{k\in S(x)} g_k(x;R)\,E_k(x)$. Here $x$ is a token's hidden state, $S(x)$ is its selected top-$k$ expert set, $g_k(x;R)$ is the gate weight produced by router parameters $R$, and $E_k(x)$ is expert $k$'s output. The ordinary backward pass already produces $\delta(x)=\partial L/\partial y(x)$, the loss gradient arriving at the layer output. We cache $\delta$ and the selected expert outputs. From them we define a local surrogate of the loss as a function of router weights $R'$, with downstream layers held at their linearization:

$$\tilde\ell(R')=\sum_x \Big\langle \delta(x),\ \sum_{k\in S(x)} g_k(x;R')\,E_k(x)\Big\rangle .$$

At $R'=R$, the gradient of $\tilde\ell$ equals the true router gradient. The router-SAM update applies the gradient of $\tilde\ell$ evaluated at $R+\rho\,\nabla\tilde\ell/\|\nabla\tilde\ell\|$ to the original weights, where $\rho$ is the perturbation radius. Evaluating that gradient requires only re-running the router's small matrix product and softmax on cached inputs, never the experts or later layers. Because every layer's $\delta$ comes out of one backward pass, all routers are perturbed in the same step.

The surrogate deliberately captures only the curvature created by the gating nonlinearity. This includes how the softmax trades probability mass between selected and unselected experts, which is exactly the mechanism behind runaway allocation. It ignores curvature from downstream layers. Whether that omission matters is an empirical question, and one ablation below tests it directly. The perturbation is scaled per router row as in ASAM, which makes perturbations proportional to weight magnitude. Router row norms differ widely across experts, and a fixed-radius ball would over-perturb small-norm experts.

Diagnostic phase. Before any method claims, we measure whether router sharpness actually precedes collapse. We train small Switch-style MoE language models under standard load balancing and under reduced balancing that deliberately induces collapse. At checkpoints we estimate, by power iteration on Hessian-vector products, the top eigenvalue of the loss Hessian restricted to router weights and restricted to expert weights. We log these alongside three collapse measures: per-layer load entropy, the fraction of experts below 1% of tokens, and pairwise expert-output similarity on shared tokens, the metric behind the "up to 99% similarity" finding in OMoE.

This phase motivates the method only if router curvature rises before load entropy drops. If router sharpness lags collapse or is uncorrelated with it, the premise fails and we stop. We also measure a "flip rate", the fraction of tokens whose top-$k$ set changes under a router perturbation of norm $\rho$. This gives a direct handle on allocation fragility.

Positioning. The mechanism story comes from X-MoE, which traces representation collapse to the Jacobian of dot-product routing. It counters that effect with a low-dimensional hypersphere router and a learnable temperature. Temperature controls how peaked routing probabilities are, but not how sensitive the loss is to router weights, and router-SAM targets the second quantity.

Three methods in the file attack the redundancy side of collapse. OMoE alternates normal steps with updates projected away from peer experts. It pays about 1.38× memory, and its benefit shrinks as expert count grows. Advancing Expert Specialization for Better MoE adds an orthogonality loss on expert outputs and a variance loss on gates. φ-Balancing replaces noisy per-batch balancing with a population-level objective tracked by a moving average of expert load. All three use plain first-order optimizers and never look at curvature, so router-SAM can be combined with each rather than only competing against them.

On the efficiency side, SAF/MESA avoids the second pass through trajectory-matching penalties. It was studied only in dense vision models, and MESA still costs about 15%. Bi-LoRA avoids the second pass with a parallel adversarial adapter at about 110% step time. Neither targets routing. Timing ablations draw on Towards Understanding Sharpness-Aware Minimization, which finds that SAM applied only in late training recovers most of its benefit. That paper also finds that small-batch perturbations generalize better, so we compute $\rho$-normalization per micro-batch.

Baselines

The naive reference point is an MoE trained with AdamW and a standard auxiliary load-balancing loss, with its coefficient tuned on the same budget as every other method. Any method that fails to beat this on both collapse metrics and downstream quality has no claim.

The strongest existing collapse method is φ-balancing. On fine-tuning of Moonlight-16B-A3B, φ-Balancing reports Math500 accuracy rising from 64.8% to 66.2% and HumanEval accuracy from 73.2% to 75.9%. It is the strongest competitor because it already removes the batch-noise bias that causes homogenization, at negligible cost. We reimplement it in our training pipeline, on the same data and number of steps, with an equal hyperparameter sweep, and first check that we reproduce its gain over the baseline. Beating it means that router-SAM, alone or stacked on φ-balancing, exceeds the reproduced φ-balancing result on Math500 and HumanEval by more than seed-to-seed variation over three seeds. It must also do so with lower expert-output similarity, at under 5% extra step time.

For the redundancy side we also run the orthogonality-plus-variance losses of Advancing Expert Specialization under the same budget. The file gives no per-benchmark Math500 or HumanEval number for that method. We also run X-MoE routing in the pretraining setting. There, X-MoE reports masked-language-model validation perplexity of 18.72 against 19.02 for a standard sparse MoE. We treat those numbers as a reproduction target, not a direct comparison, unless our configuration matches theirs.

The efficiency ceiling is full SAM over all parameters, in the style of SAM. We run it both for the whole run and for only the final 10% of steps. Both are compared at matched wall-clock time as well as matched step count.

Three ablations isolate the key ingredients. Exact router-only SAM perturbs the routers and redoes the full forward and backward pass. It tests whether the cached-downstream linearization loses anything. Random-direction router perturbation of the same norm, essentially noisy gating, tests whether the adversarial direction matters. Router-SAM without ASAM-style scaling tests the scale adaptation.

Why not just…?

Why not just tune the balancing coefficient or routing temperature? These cost nothing and are the first thing any practitioner tries. The evidence in OMoE and Advancing Expert Specialization is that stronger balancing prevents dead experts but produces near-identical experts, so a single coefficient trades one failure for the other. This is the most serious threat to the proposal, however. For small $\rho$, SAM approximately adds a gradient-norm penalty, and on a softmax router that could behave much like a temperature or entropy regularizer. We settle it by sweeping temperature and an explicit router-entropy penalty jointly with the balancing coefficient, at a larger tuning budget than router-SAM receives. We then check whether any point on that sweep matches router-SAM's pair of dead-expert fraction and expert similarity. If one does, the method adds nothing and we will report that.

Why not just run full SAM, perhaps only late in training? Full SAM roughly halves throughput; SAF reports SAM-family methods at 50–60% of normal speed, and that runs against the reason for using sparse models. Late-only SAM is a genuine competitor, since Towards Understanding SAM shows the final 10% of training captures most of the benefit. Late switching cannot, however, undo experts that already died early. The matched-wall-clock comparison decides between the two. It also shows whether flatness helps MoEs through the experts, in which case full SAM wins on quality, or through the router, in which case router-SAM matches it at far lower cost.

Why not just apply logits-SAM to the output head, or use MESA? Both are cheap sharpness methods that already exist, and a skeptic could argue that flattening the output end is enough. Neither touches routing: logits-SAM perturbs only the head, and MESA's trajectory penalty acts on predictions. If router flatness is the active ingredient, both should improve perplexity somewhat but leave the dead-expert fraction and flip rate unchanged. We run both as baselines to check this prediction.

Why not just use hard assignment routing such as BASE layers? BASE Layers guarantees equal load by solving an assignment problem over the whole batch. This does not stop experts from becoming redundant. It also creates a mismatch at autoregressive inference, where routing must be greedy. We include it only in the pretraining comparison, measuring expert similarity and the gap between perplexity under batch-level and greedy routing.

Proposed Experiments

The first stage is the diagnostic study on small MoE models, 8 to 64 experts per layer. They are trained with masked language modeling on the CCNet plus Wikipedia setting used by X-MoE and with causal language modeling, and we track router and expert Hessian eigenvalues against collapse onset. The second stage is pretraining comparisons in the same setting. These report validation perplexity, load entropy, dead-expert fraction, expert-output similarity, flip rate, and measured step time. We scale expert count to test whether router-SAM avoids the degradation OMoE shows at high expert counts. The third stage fine-tunes the pretrained MoE models on GLUE and SQuAD 1.1, where OMoE reports average GLUE gains of +0.62 to +0.9. We include the small tasks MRPC and CoLA, where OMoE degrades, as a robustness check. The fourth stage fine-tunes Moonlight-16B-A3B on math and code data, evaluated on Math500 and HumanEval against reproduced φ-balancing.

Three checks are decisive. First, router curvature must lead collapse in the diagnostic runs. Second, cached-surrogate router-SAM must reach at least most of the gain of exact router-only SAM; if the linearization loses most of the benefit, the cheap version is not worth publishing. Third, router-SAM must separate from the best point on the temperature, entropy, and balancing sweep on both allocation health and downstream accuracy, at a measured overhead within the 5% target. Failure on any one of these tells us which part of the story is wrong, and we would report it as such.