Expert Collapse as Winner-Take-All Singular Value Growth in the Router

Mixture-of-experts language models replace one large feed-forward block with many smaller "experts." A small learned layer, the router, sends each token to a few of them. These models often fail early in training through expert collapse: the router sends most tokens to a handful of experts and the rest stop learning. This note proposes that a large part of this collapse is a spectral event inside the router's weight matrix. The router's top singular direction aligns with the average token, which makes some experts look better for every token at once. Theory from deep linear networks predicts exactly this kind of runaway growth of a leading direction.

That view yields a sharp, two-sided test. Factorize the router into a product of $L$ linear matrices. The product folds back into one matrix after training, so inference cost does not change. Depth then speeds up winner-take-all growth of the router's singular values. If the account is right, a deeper router without any balancing should collapse faster, not slower. If the "average token" channel is removed by centering the router's input, the same depth should instead act as an adaptive low-rank prior on the directions that actually tell tokens apart. The result would be specialized, balanced routing without a fixed rank hyperparameter or a heavily weighted auxiliary loss. Collapse and specialization would become two outcomes of one dynamic, steered by one architectural choice.

Core Claim

Write the router as $s(x) = W x$, where $x \in \mathbb{R}^d$ is a token's hidden state, $W \in \mathbb{R}^{K \times d}$ maps it to one score per expert for $K$ experts, and the top-$k$ scores pick the experts. Split each token into the running mean token $\mu$ and its deviation $z = x - \mu$:

$$ s(x) = W\mu + W z . $$

The first term is identical for every token, so it works as a learned per-expert bias. Any growth of $W$ along $\mu$ pushes all tokens toward the same experts. That is load collapse. The second term is what lets the router tell tokens apart, and its useful structure is plausibly low-rank. L2R obtains strong gains with a routing space of rank 2.

Implicit Regularization in Deep Matrix Factorization shows what happens when a matrix is factorized as $W = W_L \cdots W_1$ and trained by gradient flow from small, balanced initialization. Each singular value $\sigma_i$ of the product then grows as

$$ \frac{d\sigma_i}{dt} \propto \sigma_i^{\,2 - 2/L}, $$

so for $L > 2$ directions that start slightly larger pull away from the rest. The leading direction of a plain router (whose input is the residual stream) will tend to sit near $\mu$, because token representations share a large common component. We therefore predict three things. First, deep factorization speeds the growth of the $\mu$-aligned direction and the collapse that follows. Second, input centering cuts that channel. Third, after centering, depth chooses the routing rank adaptively, as IRMAE recovers the intrinsic rank of synthetic data without being told it.

The theory applies more cleanly here than in most transfers of deep-linear results. Inside the factorized router there are no skip connections and no nonlinearities. The Low-Rank Simplicity Bias in Deep Networks identifies skip connections as what protects rank in residual networks, and the router chain has none.

Relation to Prior Work

Three papers already touch both directions, and none of them tests this hypothesis.

L2R supplies the strongest motivating evidence. It projects tokens through an explicitly low-rank matrix, adds a bounded scoring function, and uses multiple anchors per expert. Its rank is fixed by hand, and it does not study how router optimization creates or destroys rank. We replace the fixed rank with depth-induced rank and add a mechanism for why collapse happens.

Eigenvectors of Experts are Training-free Non-collapsing Routers computes singular vectors of trained expert weights and routes with them after training. It observes that these vectors stay orthogonal while learned router embeddings become correlated in deeper layers. That observation is consistent with our claim that the defect lives in the router's spectrum and not in the experts. However, their method cannot repair dead experts that already formed during pretraining, and expert dropping hurt GSM8K.

Beyond Parameter Count studies an implicit bias of soft mixing toward discoverable experts as the expert count grows. That bias comes from width and soft mixing, not from depth-driven singular value dynamics.

Two other papers constrain the design. OMoE and Advancing Expert Specialization show that strong balancing pressure causes the opposite failure, homogenization: up to 99% similarity between expert outputs. Any fix must therefore be judged on load and on expert diversity together.

Two papers motivate the early-phase focus. OLMoE reports that routing decisions saturate within roughly the first 1% of training. Scaling Vision with Sparse Mixture of Experts describes the rich-get-richer feedback loop. Our account names a specific channel for that loop.

Research Program

The first phase is measurement on standard routers, before any intervention. During small-scale MoE pretraining we will log several quantities at every MoE layer:

  • the router's singular value spectrum and effective rank, defined as the entropy of the normalized singular values following Huh et al.;
  • the cosine between the router's top right-singular vector and the running token mean;
  • the fraction of logit variance that is token-independent, meaning the $W\mu$ term;
  • the load distribution, measured as maximum load share, entropy, and dead-expert count;
  • the pairwise similarity of expert outputs on shared tokens, which tracks homogenization.

The mechanistic claim is falsified if load concentration does not follow growth of the $\mu$-aligned component in time and across layers. This phase also addresses the apparent disagreement between OMoE and SSMoE. We will compute expert weight-space orthogonality and output-space similarity on the same checkpoints to see whether they can coexist.

The second phase is a controlled intervention crossing two factors:

  • Router depth. $L \in \{1, 2, 3, 4\}$, with small balanced initialization of the factors.
  • Router input. Raw input versus centered input, where centering subtracts an exponential moving average of the router input. Its cost is a single vector per layer.

Each setting is run with and without a conventional balancing loss. The theory predicts an interaction. Without centering, collapse should arrive earlier as $L$ grows. With centering, collapse should not worsen with depth, and the effective routing rank should fall to a data-dependent value that differs across layers. Deeper MoE layers should settle at lower rank if their activations are already compressed by network depth, the question raised by Implicit Bias and Loss of Plasticity in Matrix Completion.

The third phase is a synthetic check where ground truth is known. Tokens are drawn from a mixture with a controllable shared mean and a known number of clusters, and an oracle routing is fixed. This tests whether the deep centered router recovers the cluster subspace's rank and whether the uncentered router collapses at the rate the singular value equation predicts.

Baselines

Naive reference. A standard single-matrix linear router with a Switch-style batch load-balancing loss, plus the same router with no balancing loss to expose raw collapse. L2R reports this linear router at 22.7% MMLU after 10B tokens on the OLMoE recipe (L2R). That gives a reproduction target for our harness.

Strongest existing method. L2R itself, which reports 27.6% MMLU in the same 10B-token OLMoE setting and 76.58% top-1 on ImageNet-1K with ViT-S. It is the closest competitor because it already exploits low-rank routing and Lipschitz control. Any gain we show could be L2R's gain under another name.

To make the comparison fair, L2R gets a rank sweep over $r \in \{2, 4, 8, 16\}$, matched to our sweep over $L$ and centering. Both methods get the same tokens, model size, optimizer, and number of tuning runs. We will also run our depth factorization in front of L2R's bounded scoring function, so the two contributions can be separated rather than only competed. Beating L2R means matching or exceeding its tuned MMLU and perplexity at equal tuning budget while also showing lower expert-output similarity. Equal accuracy without the rank hyperparameter counts only as a partial result, and we will report it that way.

Loss-based competitors. Two explicit-loss methods are rerun at our scale with their prescribed loss weights and an equal tuning budget:

Their published numbers come from a scale we will not reach and are not directly comparable. We will not claim to beat them at their scale.

Ablations. Two ablations isolate the key ingredient: depth without centering, and centering with $L = 1$. If centering alone delivers the whole benefit, depth adds nothing and we will say so. A nuclear-norm penalty on a single-matrix router serves as the explicit counterpart to depth's implicit penalty.

Why not just…?

Why not just sweep L2R's rank? This is the most serious threat. A fixed rank tuned per model may capture most of what adaptive rank offers. The matched-budget sweep above settles it. Depth has a structural advantage only if the best rank differs across layers or shifts during training, which a single global $r$ cannot follow. Phase two measures exactly this. If the per-layer ranks learned by the deep router are nearly uniform and tuned L2R matches it, the method's contribution collapses into the diagnostic and the centering trick, and the paper will be framed that way.

Why not just center the router input and stop there? Centering is the cheapest consequence of our account and may carry most of the load-balance benefit. The $L = 1$ centered ablation tests it directly. Depth earns its place only if it adds measurable specialization beyond centering alone: lower expert-output similarity and lower routing rank at equal perplexity.

Why not just add a balancing or orthogonality loss? These losses work at scale, as the cited results show. Our measurements test whether they fix the cause or only the symptom. If a balancing loss holds load even while the $\mu$-aligned component keeps growing, the router is effectively fighting a learned bias. That predicts the homogenization reported by OMoE. The experiments compare balance and expert similarity jointly, so a loss that balances load by making experts alike will show up.

Why not just route post hoc with expert eigenvectors? SSMoE works on frozen models. It cannot revive experts that received no gradient during pretraining, and its pruning hurt math reasoning. We include it as a post-hoc layer on our checkpoints. If centered deep routers make it unnecessary, the gap between learned and spectral routing should shrink, and that is a measurable outcome.

Proposed Experiments

The main language setting follows L2R's protocol: OLMoE-style models trained on a 10B-token subset of the OLMoE pretraining mix, evaluated on held-out perplexity and MMLU. We will add GSM8K, given SSMoE's finding that math is the most capacity-sensitive task. No reported result was found on Lacuna for load-entropy or dead-expert counts in this exact setting, so our harness establishes those numbers for every baseline. The vision setting is ViT-S MoE on ImageNet-1K, against L2R's 76.58% top-1 and 91.50% top-5. The synthetic mixture task described in phase three runs alongside both.

Decisive check 1: direction of the depth effect. In uncentered, unbalanced runs, does time-to-collapse shorten monotonically as $L$ grows, and does centering reverse the trend? A depth that helps uniformly, regardless of centering, would contradict the mechanism even if it improved accuracy.

Decisive check 2: rank adaptivity. In the synthetic task, does the effective rank of the centered deep router match the number of planted clusters minus one across several planted values? In the language setting, does that rank vary across layers in a way a single fixed rank cannot match?

Decisive check 3: the downstream bar. At matched tuning budget, does the centered deep router equal or beat tuned L2R on MMLU and perplexity while showing lower expert-output similarity than both explicit-loss baselines?

Passing the first two with only a tie on the third would still establish a mechanistic account of collapse, with a zero-inference-cost control knob. Failing the first would mean the singular value account of collapse is wrong, and we would report that.