Weight Decay Keeps Duplicate Experts Alive; Relevance Priors Consolidate Them

A mixture of experts is a network that sends each input to a few specialized sub-networks ("experts") picked by a small gating model. Practitioners routinely give these models more experts than the data needs. Statistical theory shows this over-provisioning has a hidden cost. The model's predictions stay accurate, but the experts' own parameters are learned far more slowly, sometimes at rates like $n^{-1/12}$ or $1/\log n$ instead of the usual $n^{-1/2}$. The reason is that surplus experts end up as near-copies that split one job between them, and the loss cannot tell how the job is divided. This note argues that the choice of weight penalty decides what happens to those copies. Ordinary weight decay actively pushes duplicates toward an even split. That even split is the configuration theory identifies as worst. A group penalty is indifferent to the split. An automatic relevance determination (ARD) prior is a Bayesian prior that gives each expert its own learned "should this exist?" scale. It strictly prefers handing the whole job to one copy and switching the others off, which it does during training.

Put in two sentences: overspecified experts sit in a flat valley of equivalent ways to share work, and the penalty's curvature along that valley picks where training lands. Squared penalties roll toward the balanced, slowest-converging point. Concave per-expert penalties of the ARD kind roll toward a sparse, exactly-fitted point, where fast parameter convergence is known to return. We propose to prove this in a shallow mixture of experts, test it against the rates in the theory literature, and check whether it survives in small sparse language and vision models trained from scratch.

Core Claim

Consider two experts that duplicate one true expert. Suppose they share its output in proportions $t$ and $1-t$ through their output-layer weights, which is possible whenever gates are independent (sigmoid gating) or expert outputs can be rescaled. Let $c$ be the squared norm of the true expert's output weights and $D$ the number of those weights. The three penalties then cost

$$ R_{\text{wd}}(t)=c\,[t^2+(1-t)^2],\qquad R_{\text{group}}(t)=\sqrt{c}\,[t+(1-t)],\qquad R_{\text{ARD}}(t)=\tfrac{D}{2}\big[\log(t^2c+\epsilon)+\log((1-t)^2c+\epsilon)\big]. $$

Here $R_{\text{wd}}$ is weight decay and $R_{\text{group}}$ is a group lasso on each expert. $R_{\text{ARD}}$ is the penalty obtained by integrating out a per-expert precision $\alpha_k$ in the Gaussian prior $W_k\sim\mathcal N(0,\alpha_k^{-1}I)$ under a near-flat hyperprior, with $\epsilon$ a small floor.

Weight decay is minimized at $t=1/2$. The group penalty is constant. The ARD penalty is minimized at $t\to 0$ or $1$. This is elementary, but it has a sharp consequence. Khamaru et al. show that balanced overspecified mixtures, where redundant weights are equal, degrade estimation most. So the default AdamW-style penalty steers sigmoid-gated experts into the worst regime.

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts recommends sigmoid gating because it avoids the softmax's forced competition. That recommendation is derived for unpenalized maximum likelihood with a fixed number of experts. Our claim is that the recommendation and the penalty interact. Sigmoid gating combined with an ARD prior should both prune surplus experts and reach the exactly-fitted regime, where Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts proves $n^{-1/2}$ parameter rates. Sigmoid gating combined with weight decay should do neither.

The softmax case is different. There the gate shares sum to one, so duplicates split gate mass rather than output magnitude, and weight decay no longer favors balance. We therefore predict that the direction of the penalty effect depends on gate type. This is a falsifiable, non-obvious prediction.

Relation to Prior Work

The motivating evidence comes from convergence theory for overspecified mixtures.

All three analyze the unregularized estimator with a fixed expert count. We instead ask which point in the flat valley a penalized trainer selects.

The closest method is the Dendrogram Selection Criterion (Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts; Model Selection for Gaussian-gated Gaussian Mixture of Experts). It trains one overspecified model, clusters its experts afterwards, and merges them, provably restoring $n^{-1/2}$ rates. It is post-hoc, it requires dense soft gating, and it leaves training inside the slow regime. Our proposal consolidates during training and applies to hard top-$k$ routing, because the relevance scale multiplies whatever expert is selected.

The ARD machinery itself is reused, not invented. Propagation Algorithms for Variational Bayesian Learning supplies per-column precisions that go to infinity for unused factors. Efficient Network Automatic Relevance Determination supplies surrogate updates that avoid cubic covariance inversions. Neither work touched routed architectures, and neither framed ARD as selecting among equivalent duplicate solutions rather than removing irrelevant ones.

Expert-pruning work for large sparse models constrains the design. The methods covered are:

All of them take the expert count as a budget or require a trained checkpoint. None asks why redundant experts formed.

Research Program

The first part is analysis in a single-layer mixture of experts with ReLU two-layer experts, under both sigmoid and softmax gating. We will characterize the stationary points of the penalized likelihood along the duplicate-splitting directions. The first target is a proof that under sigmoid gating the ARD-penalized estimator has, with high probability, no balanced duplicate stationary point that is a local minimum. The second target is that its surviving experts inherit the exactly-fitted rate from the top-$k$ analysis of Nguyen et al.. A third target is a matching negative result: weight decay yields a balanced local minimum.

We will also bound over-pruning. This means showing that a true expert whose gate mass and signal exceed a threshold set by $\epsilon$ and $n$ survives. This is exactly the failure mode that ARD's marginal-likelihood optimization is known to exhibit under misspecification.

The second part is the algorithm. Each expert carries a precision $\alpha_k$ on its output-layer weights and on its gate's scalar magnitude. The precisions are updated in closed form every few hundred steps with a diagonal approximation of the posterior spread, following the surrogate spirit of Network ARD. When $\alpha_k$ exceeds a threshold, expert $k$ is removed from the router's candidate set, which frees its memory. For top-$k$ routing, the precision acts on a continuous multiplier applied after selection, so the discontinuous routing boundary never has to be differentiated.

The third part scales the method up. We pretrain small sparse language models from scratch and fine-tune vision mixtures, then ask whether consolidation happens, whether it harms quality, and whether it reaches memory savings comparable to post-hoc pruners without a budget being supplied.

Baselines

The naive reference point is unpenalized training of the overspecified model with the same number of experts and no pruning. It measures how slow the flat valley is. Next to it sits the oracle: the same architecture trained with the true expert count, which bounds what any consolidation method can achieve.

The central ablation is weight decay versus group lasso versus ARD. All three use identical architecture, optimizer, data, and step count, and each penalty strength is tuned over the same grid of eight values on held-out likelihood. Only the shape of the penalty changes, so this comparison isolates the proposal's key ingredient. It is run separately for sigmoid and softmax gates to test the gate-type prediction.

The strongest competitor on final parameter recovery is the Dendrogram Selection Criterion (Hai et al.). It is the only method in the file with a proven restoration of $n^{-1/2}$ rates after merging, which Thai et al. report as an empirical $N^{-1/2}$ post-merge rate on synthetic data. On real data, the file reports that it picked 2 experts on maize proteomics where AIC picked 18. We run it on the same unpenalized overspecified checkpoint our methods start from, using the authors' dissimilarity. Beating it means matching its final rate while reaching a given parameter error in fewer samples or steps, and working on top-$k$ routers where it does not apply. Merely matching it at the end would still count as a partial result.

The strongest gate-design competitor is sigmoid gating with no penalty (Nguyen, Ho, and Rinaldo). It reports empirical Voronoi loss decaying as $n^{-0.51}$ for ReLU experts, versus $n^{-0.24}$ for softmax. We reproduce their synthetic setup exactly. Beating it means matching its rate while removing the surplus experts.

At scale, the strongest post-hoc pruner is DiEP (Bai et al.). On Mixtral 8x7B at 50% expert sparsity it reports retaining 92% of original performance, with 1.28x speedup and 48% memory savings. We have no reported result found on Lacuna for DiEP on small from-scratch models, so we will run it ourselves on our weight-decay checkpoint. We give it the sparsity level our ARD run discovered and its recommended 128 to 1024 C4 calibration samples. Beating it means equal or lower perplexity at that sparsity without calibration, and without being told the budget.

In vision, router-norm pruning (A Provably Effective Method) reports removing 75% of experts on CIFAR-10 within 1% of unpruned accuracy. We match its fine-tuning protocol.

Why not just…?

Why not just use sigmoid gating, which already restores fast rates? This is the most serious threat on the estimation question. Nguyen, Ho, and Rinaldo show $n^{-1/2}$ rates for active overspecified parameters under unpenalized maximum likelihood. Their result keeps every surplus expert in memory, however. Our core claim is that the realistic trainer, which applies weight decay, pushes sigmoid-gated duplicates toward balance. If the ablation shows sigmoid plus weight decay tracking the $n^{-0.51}$ curve and not drifting toward balanced splits, the rate argument for ARD collapses. What would then remain is the memory benefit, and the proposal should be reframed around that.

Why not just train normally and merge experts afterwards with dendrograms? The Dendrogram Selection Criterion is provably correct at the end of training. It cannot shorten the time spent in the slow regime, and it assumes dense soft gating. We will plot parameter error against training step for both methods. If dendrogram merging at every checkpoint matches ARD's trajectory, in-situ consolidation adds nothing on soft-gated models, and our contribution narrows to hard top-$k$ routing.

Why not just add weight decay or a group lasso on each expert? The display above answers this directly. Weight decay prefers the balanced split, and the group lasso is exactly flat along it, so neither breaks the degeneracy. Convergence Rates for Softmax Gating Mixture of Experts also indicates that such penalties do not decouple gate from expert derivatives. The penalty-shape ablation tests this empirically rather than by assertion.

Why not just drop the least-used experts? Balanced duplicates each receive about half of one true expert's traffic. That makes them look moderately used, not idle, so frequency counts will miss them. Shapley-MoE and A Provably Effective Method also both report that token counts discard rare but essential experts. We include frequency pruning at the ARD-discovered sparsity as a reference.

Proposed Experiments

The synthetic study reproduces the setups of Nguyen, Ho, and Rinaldo and Thai et al.. It uses a true expert count of 2 to 4 and fits 2 to 4 times as many experts, with ReLU and linear experts. Sample sizes span $10^3$ to $10^6$, with 20 seeds each. We measure four things: the Voronoi parameter loss and its fitted log-log slope; the recovered expert count; the duplicate split fraction $t$ over training; and steps to reach a fixed parameter error.

The decisive checks are three. First, under sigmoid gating, weight decay should drive $t$ toward $1/2$ and ARD should drive it to 0 or 1. Second, under softmax gating this contrast should shrink or vanish. Third, ARD's slope should approach $-0.5$ with the correct expert count recovered. A misspecified variant, in which the truth is a smooth non-mixture function, tests over-pruning. This addresses the open problem the theory papers leave unresolved. We will report how many experts survive as $n$ grows and whether held-out likelihood suffers.

The scale study pretrains roughly 100M to 400M active-parameter sparse transformers on FineWeb with 32 or 64 experts per layer and top-2 routing. It compares weight decay, weight decay followed by DiEP, and ARD. We report WikiText-2 perplexity, MMLU and ARC accuracy, expert memory, and the discovered expert count per layer. The per-layer count tests whether ARD rediscovers the depth-dependent redundancy DiEP reports, with deeper layers pruning more.

A vision replication fine-tunes a small vision mixture on CIFAR-10 and CIFAR-100 against router-norm pruning. The result that would most change our view is ARD pruning comparable numbers of experts at scale while showing no measurable balancing under weight decay. That outcome would mean the benefit is compression alone, and the convergence story does not transfer beyond shallow models.