Mixture-of-experts language models send each token to a few of many small feed-forward subnetworks ("experts"). The model that makes this choice, the router, often settles on a handful of favorites, and the remaining experts go idle. This is called expert collapse. The usual explanation is a rich-get-richer loop: experts that receive more tokens improve faster and so get picked more. This proposal points to a second force that the field has not examined. Weight decay, the standard penalty that shrinks every weight a little at each step, is applied equally to every expert. The learning signal an expert gets, however, scales with how many tokens it receives. A rarely chosen expert therefore faces strong shrinkage and weak signal. Work on implicit low-rank bias shows that this ratio of shrinkage to signal sets how many independent directions a weight matrix keeps, which is its effective rank. Our claim is that uniform weight decay quietly reduces the effective rank of under-used experts, which pushes them toward uselessness. We further claim that the same mechanism explains why well-used experts end up in nearly orthogonal subspaces.
The work has two parts. The first tests a quantitative prediction that an expert's effective rank grows with the square root of its token share. The second evaluates a one-line training change we call load-matched weight decay, which scales each expert's decay by its running token share. Under this change, every expert sees the same decay-to-data ratio it would see if trained alone on its own tokens. If the change lowers collapse rates as much as heavier machinery such as subspace-manifold routers or orthogonality losses, practitioners get a nearly free fix. If it does not, we learn that collapse is driven by routing dynamics alone. Either result clears up a confused part of the literature.
Core Claim
The low-rank bias literature supplies the mechanism. Low-rank bias, weight decay, and model merging in neural networks proves that at stationary points of a weight-decayed loss, a network's "pseudo-rank" (a weighted average of the stable ranks of its layers) is bounded by $O(\sqrt{1/\lambda})$, where $\lambda$ is the decay coefficient. Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networks shows that decay acts as an active force that removes weight mass outside the subspace the gradients span.
Now consider expert $e$ receiving a fraction $p_e$ of tokens. Under plain gradient descent, its data gradient is proportional to $p_e$, while its decay term is not. The expert therefore behaves like a standalone network trained with an effective coefficient
$$ \lambda_{\text{eff},e} = \lambda / p_e, \qquad \text{so} \qquad \text{rank}_e \lesssim C\sqrt{p_e/\lambda}, $$
where $C$ is a constant independent of the expert. The square-root form comes directly from the cited $O(\sqrt{1/\lambda})$ bound.
Under AdamW the gradient scale is normalized away. The ratio still matters, though, through the signal-to-noise ratio of each expert's gradient, which falls as fewer tokens are averaged. We therefore treat the exponent as something to measure rather than assume. The same cited paper's "subspace residency" result supplies the second half of the claim: when decay acts on models trained on near-orthogonal data, each model ends up in a narrow, non-interfering subspace. Routing splits the data, so experts that receive enough tokens should drift into separate subspaces.
This would explain why Eigenvectors of Experts are Training-free Non-collapsing Routers finds naturally orthogonal expert eigenvectors in trained checkpoints. It would also explain why Advancing Expert Specialization for Better MoE and Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer report up to 99% similar experts under strict load balancing. In that setting the experts see mixed, non-partitioned data, and residency predicts shared rather than separate subspaces.
Relation to Prior Work
The papers closest to this intersection all apply low rank deliberately and only to the router. On the Representation Collapse of Sparse Mixture of Experts shows that routing gradients pull hidden states into the span of the router's expert embeddings, and fixes this with a rank-16 routing projection plus normalization. L2R routes in a latent space with rank as low as 2. Neither studies the rank of the expert weights themselves, or what the optimizer's decay does to them. Grassmannian Mixture-of-Experts represents each expert as a subspace and enforces orthogonality between experts with an explicit loss and variational inference. We ask whether ordinary weight decay, rebalanced by load, produces that same geometry without the extra machinery.
SSMoE motivates our study, since it found the orthogonality but offered no explanation and only uses it after training. Weight decay induces low-rank attention layers supplies the method precedent. It shows that a single decay value is suboptimal and that setting decay separately per module group improves perplexity. We go further by making decay depend on load rather than on module type. Balancing the Experts studies factorized LoRA experts but treats their low rank only as a way to save compute. We leave that reinforcement learning setting as a later extension.
Research Program
The diagnostic phase trains small mixture-of-experts language models with top-k token routing and the standard auxiliary balancing loss. For each run we log, per expert and over time, five quantities: the running token share, the stable rank and nuclear norm of the expert's weight matrices, the norm of its row in the router, its gradient signal-to-noise ratio, and the principal angles between its top singular subspace and those of the other experts. Across a grid of decay values, we fit log stable rank against log token share. The SGD analysis predicts a slope of 0.5, and we measure whether AdamW matches it. We also check whether experts that later die first lose rank before they lose tokens. That ordering would separate decay-driven starvation from pure routing feedback.
The intervention phase replaces uniform decay with $\lambda_e = \lambda \cdot E\,\bar p_e$. Here $E$ is the number of experts and $\bar p_e$ is an exponential moving average of the expert's token share, so an expert at the average load keeps the base value. We decouple the router's decay from the experts' decay and study the two separately. The router-side question is genuinely open. Shrinking a cold expert's router row could either suppress that expert or rescue it, depending on the sign of its logits. The norm-dominated competition described in L2R suggests router norms matter, but we have no prior evidence on the direction.
Baselines
The naive reference is the standard recipe: top-k routing with a Switch-style auxiliary balancing loss and a single uniform AdamW decay value. This is the setting in which Grassmannian Mixture-of-Experts reports that baselines collapse in 20–65% of seeds.
The strongest existing method on our main metric is GrMoE, which reports a 0% collapse rate across seeds at 350M, 1.3B and 2.7B parameters on OpenWebText. The file gives no numeric perplexity for it, so we will reproduce it ourselves on identical data and token budgets. Beating it means matching its 0% collapse rate across at least five seeds while reaching equal or lower validation perplexity, at lower wall-clock cost.
For downstream accuracy we compare against L2R, which reports MMLU rising from 22.7% for a linear router to 27.6% after 10B OLMoE tokens. Since our change acts on the experts while L2R acts on the router, we also test the two combined. We further include $L_o+L_v$ from Advancing Expert Specialization for Better MoE and φ-Balancing, each implemented in the same codebase.
The ablations isolate the key ingredient, which is coupling decay to load. The first sets decay to zero on the experts. The second tunes separate router, expert and attention decay values but keeps them load-independent, following Weight decay induces low-rank attention layers. That paper's best dense result is a perplexity of 11.52 on the Pile, which shows how much per-group tuning alone can buy. The third scales decay by a random permutation of the load vector, so the total decay is unchanged but the alignment with load is destroyed.
Every method gets the same data, tokens, model size and a tuning budget of an equal number of trials over its own hyperparameters.
Why not just…?
Why not just turn off weight decay on the experts? This is the most serious threat. If decay is what starves cold experts, removing it might do as well as load-matching. The low-rank literature predicts a cost. Towards Better Generalization finds that without decay the network keeps high-rank, noise-memorizing weights. Weight Decay Improves Language Model Plasticity finds that decay preserves the ability to adapt during later fine-tuning. Removing decay should also erase the subspace separation that residency predicts. We will measure all three: collapse rate, perplexity, and downstream fine-tuning gains. If zero decay ties load-matched decay on every measure, we will report the simpler conclusion that decay should be dropped on experts.
Why not just tune the decay values separately per module group? Per-group tuning gives one value to all experts, so it cannot correct the imbalance between hot and cold experts. That imbalance is what our mechanism targets. The load-independent group-tuning ablation and the random-permutation control test exactly this.
Why not just use expert-choice routing, which equalizes load by construction? Mixture-of-Experts with Expert Choice Routing lets each expert select a fixed number of tokens, which removes load imbalance entirely. It also requires seeing a whole batch of tokens before routing, which complicates token-by-token generation. We use it as a mechanistic control rather than a competitor. Our account predicts that under expert choice, the relationship between rank and load disappears and uniform decay becomes harmless. If load imbalance persists in rank anyway, our mechanism is wrong.
Why not just apply SSMoE after training? Post-hoc spectral routing cannot revive experts that lost capacity during pretraining. SSMoE itself reports that pruning experts this way hurts GSM8K accuracy. We will also apply SSMoE to our checkpoints, as a readout of whether load-matched training gives better-separated spectra.
Proposed Experiments
The main testbed is OpenWebText at 350M and 1.3B parameters with five or more seeds per setting, matching the GrMoE protocol. Metrics are collapse rate, validation perplexity, and specialization statistics. We add a 10B-token run on the FineWeb data to report MMLU, HellaSwag, ARC-Easy and PIQA against the L2R number. Pairwise expert similarity and the principal angles between experts measure homogenization.
Three results are decisive. The first is the slope of log stable rank against log token share; the SGD analysis predicts 0.5, and a slope near zero falsifies the rank mechanism. The second is whether load-matched decay lowers the collapse rate below the uniform baseline while the random-permutation control does not. The third is whether, under expert choice routing, uniform and load-matched decay become indistinguishable. A further check asks whether principal angles between experts grow with routing purity under decay but not without it, which would give a mechanistic explanation for the SSMoE orthogonality finding.
References
- Low-Rank Bias, Weight Decay, and Model Merging in Neural Networks
- Towards better generalization: Weight Decay induces low-rank bias for neural networks
- Eigenvectors of Experts are Training-free Non-collapsing Routers
- Advancing Expert Specialization for Better MoE
- Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
- On the Representation Collapse of Sparse Mixture of Experts
- L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
- Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
- Weight decay induces low-rank attention layers
- Balancing the Experts: Unlocking LoRA-MoE for GRPO via Mechanism-Aware Rewards
- $\phi$-Balancing for Mixture-of-Experts Training
- Weight Decay Improves Language Model Plasticity
- Mixture-of-Experts with Expert Choice Routing