Routing Collapse Is Minority Collapse: Fixed-Frame Routers for Mixture of Experts

A mixture-of-experts model splits a layer into many parallel sub-networks ("experts"). A small linear model, the router, decides which few experts process each token. Three things go wrong in practice. Most tokens drift to a handful of experts. Different experts learn near-identical functions. Hidden states get squeezed into a thin subspace around the router's weight vectors. The usual fixes add extra loss terms whose weights must be tuned: a load-balancing loss, a similarity penalty between experts, or both. Too little weight and the model collapses. Too much and the extra gradients hurt the main task.

This note proposes to treat the router as what it literally is: a K-way classifier over experts, trained on labels it generates itself. Once framed that way, a well-developed theory applies. Neural collapse is the observation that trained classifiers drive each class's features to a single point, spread the class centers out as a symmetric, maximally separated simplex, and align the classifier weights with those centers. Its known failure under class imbalance is "minority collapse": weight vectors of rare classes fold onto each other. Our claim is that MoE routing collapse is minority collapse of the router, and that subspace shrinkage is ordinary neural collapse of the router's inputs. If that is right, the cure the imbalanced-classification literature found should transfer: freeze the classifier as a simplex. Doing so would give expert diversity and balance without auxiliary loss weights to tune.

Core Claim

We make two claims. The first is diagnostic. In a standard sparse MoE, the four neural-collapse measurements applied to the router tell us what is happening and predict it. Those measurements are within-expert variability of router inputs, how close the expert centroids are to a simplex, how well router weights align with those centroids, and whether routing reduces to a nearest-centroid rule. Concretely, rarely chosen experts should show router weight vectors that are more mutually aligned than those of popular experts. This is the signature that Inducing Neural Collapse in Imbalanced Learning and Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem identify for minority classes.

The second claim is constructive. A router whose expert vectors are frozen to a simplex equiangular tight frame, a set of K unit vectors with identical, maximally negative pairwise cosines, removes the router's ability to fold experts together. It does this in a low-dimensional routing projection, so any induced collapse stays confined there. Pairing this frozen router with the gradient-free load bias of Auxiliary-Loss-Free Load Balancing should match or beat tuned multi-loss methods. The resulting model has neither a balancing weight nor a diversity weight, and its results should vary much less across settings of its remaining knobs.

The router score for token hidden state $x$ and expert $i$ is

$$s_i(x) = \frac{e_i^\top P x}{\tau\,(\|P x\| + \epsilon)}, \qquad E = [e_1,\dots,e_K] = \sqrt{\tfrac{K}{K-1}}\;U\Big(I_K - \tfrac{1}{K}\mathbf{1}\mathbf{1}^\top\Big).$$

Here $P$ is a learned projection to a routing space of dimension $d_r \ge K-1$. $E$ is a fixed simplex frame, with $U$ a fixed random orthonormal $d_r \times K$ matrix. $\tau$ is a temperature and $\epsilon$ is a small constant. Top-$k$ selection uses $s_i(x) + b_i$, where $b_i$ is the load bias updated outside backpropagation as in Loss-Free Balancing. The gating weights applied to expert outputs use $s_i(x)$ alone.

Research Program

Diagnostic study. The first part tests the diagnosis on existing designs. We train standard linear-router sparse MoEs on enwik8 and WikiText-103 and log, across training and per layer, three things. The first is the neural-collapse quantities of Papyan et al., with "class" replaced by "expert assignment." The second is the effective rank of hidden states entering the MoE layer, which is the subspace shrinkage X-MoE attributes to dot-product routing. The third is the pairwise cosine structure of router vectors, stratified by expert load.

Three predictions could each fail. Within-expert variability of router inputs should fall steadily while effective rank falls. Low-load experts should have higher mutual cosine than high-load experts. Adding the auxiliary balancing loss should reduce this cosine gap much as class reweighting does in the imbalance literature.

A secondary question comes from SimSMoE, which reports that collapse concentrates in early layers. We will check whether early-layer routers approach the collapsed geometry faster. If so, that would be a concrete handle on the open question of how the asymmetric margin bias of Kunin et al. acts across gated branches. We will treat this as exploratory and not claim a theorem.

Frozen-frame router. The second part builds and stress-tests the router defined above. It adapts the fixed classifier of Yang et al., with three differences. That work has true labels and uses a dot-regression loss. A router has no labels, so the frame only shapes the gate, and the task loss alone trains $P$ and the experts. We add LFB's bias for selection. We add cosine normalization with the small denominator constant that the perturbed cosine router analysis shows restores polynomial parameter-estimation rates.

The design choice that most needs testing is the routing dimension. A simplex frame requires $d_r \ge K-1$, which is larger than X-MoE's 16 dimensions once there are 32 or 64 experts. If routing ran on the full hidden state, freezing the frame would force the main representation to collapse toward K directions. The projection $P$ is what keeps that collapse confined. We will verify this by checking effective rank of the full hidden state.

Hyperparameter sensitivity. The third part makes sensitivity the primary outcome, not an afterthought. For every method we sweep each of its own knobs over a pre-registered log grid. For the aux-loss baseline that is α. For SimSMoE it is α, β, and the similarity-check frequency. For LFB it is the bias update rate. For cosine routers it is τ, or its initialization if learnable. For each method we report two numbers: the spread between best and worst validation outcome, and the fraction of grid points within 1% of the method's best. Each method gets the same number of total training runs.

Baselines

The naive reference is a standard top-2 sparse MoE with a linear router and the conventional auxiliary balancing loss at its commonly used default weight, with no tuning. This anchors how much any method helps and exposes the α-sensitivity described by Wang et al.

The strongest balancing competitor is Loss-Free Balancing: a linear router with a gradient-free per-expert bias. The paper reports validation perplexity 9.50 against 9.56 for the aux-loss baseline on a 1B model, with max load violation 0.04 against 0.72. It already removes α, so our method must beat it on something other than balance. We expect the advantage on expert redundancy, hidden-state rank, and downstream quality.

The strongest diversity competitor is SimSMoE, which adds a CKA similarity penalty between experts. It reports enwik8 validation 1.08 bits per character against 1.11 for its baseline on a 135M Brainformer.

The strongest geometric competitor is the cosine router: X-MoE with its hypersphere routing, plus the perturbed cosine router. The latter reports enwik8 1.197 bits per character against 1.213 for vanilla cosine, and DomainBed average 68.37% against 66.70% with ViT-S/16. For WikiText-103 perplexity, no reported result was found on Lacuna for any of these methods.

Reported numbers come from different model sizes and codebases, so they only sanity-check our reproductions. All comparisons are run in one codebase with identical data, token budget, expert count, active parameters, and sweep budget. Beating a baseline means one of two things. Either the proposal's best configuration improves the metric beyond seed-to-seed standard deviation across three seeds, or it matches the best configuration while showing a clearly smaller best-to-worst spread under an equal tuning budget.

Three ablations isolate the key ingredient. The first uses the same projected cosine router with LFB but a learnable frame, which is X-MoE plus LFB. The second freezes the frame at random unit vectors. The third freezes a simplex frame but omits LFB, to separate geometry from bias.

Why not just…?

Why not just tune the auxiliary balancing weight? Wang et al. show a trade-off built into this approach. Any α large enough to balance load injects gradients that raise perplexity, and small α permits collapse. Our sweep reproduces this curve directly. If a single α sits on the flat part of the curve across both datasets and scales, this alternative is adequate for balance. It still would not address expert redundancy, which we measure separately.

Why not just use Loss-Free Balancing with a linear router? This is the most serious threat on load balance, because LFB already delivers near-perfect load with no tuned loss weight. What LFB does not touch is the router's geometry. A linear router with a bias can keep tokens evenly spread across experts whose weight vectors, and whose inputs, are still collapsing into a low-rank subspace. The third ablation (frozen frame without LFB) and the first (learnable frame with LFB) settle this. If LFB alone matches the frozen frame on effective rank, expert-output similarity, and downstream scores, the geometric half of our claim is wrong and we will say so.

Why not just add an explicit diversity penalty, as SimSMoE or RIDE do? These penalties work: SimSMoE's gain on enwik8 is real. But they add a weight β, a sampling frequency, and quadratic-in-experts cost. RIDE additionally needs a multi-stage pipeline. Our claim is not that such penalties fail. It is that a frozen frame reaches comparable diversity at zero added objective terms. The sensitivity comparison is the test. If SimSMoE's best-to-worst spread over β is as flat as our spread over τ, the case for the frame rests only on cost.

Why not just freeze the router at random vectors? Random vectors in high dimension are already nearly orthogonal, so a designed simplex may add nothing. This is a genuine possibility. The regime where the two differ is $d_r$ close to $K-1$, where random frames have noticeably higher maximum pairwise cosine than a simplex. We will run both at $d_r = K-1$, $2K$, and $8K$. The prediction is that the simplex advantage shrinks as $d_r$ grows. If random frozen routers match the simplex everywhere, the contribution reduces to "freeze the router in a projected cosine space." That is a simpler and still useful finding, and we would report it as such.

Proposed Experiments

We run language modeling on enwik8 and WikiText-103 with 16, 32, and 64 experts at roughly 135M total parameters, using top-2 routing in every other layer. Primary metrics are bits per character or perplexity and max load violation as defined by Wang et al. We also record effective rank of MoE-layer inputs, mean pairwise CKA between expert outputs, the four neural-collapse measures on router inputs, and routing consistency across seeds, the instability X-MoE reports. If the 135M results are positive, a single ~1B-scale confirmation replicates the LFB setup. A vision check uses a ViT-S/16 MoE on DomainBed, where domain-specific token populations create naturally imbalanced expert demand, and follows the perturbed-cosine protocol.

The decisive checks are three. First, low-load experts in unmodified MoEs must show elevated router-vector alignment. Without that, the minority-collapse diagnosis fails. Second, the frozen simplex router with LFB must beat the learnable-frame ablation on expert redundancy or hidden-state rank without losing perplexity, while showing a smaller best-to-worst spread than SimSMoE and aux-loss routing under the equal-run sweep. Third, the simplex must beat random frozen frames at least when $d_r$ is near $K-1$. If the second check fails and the third passes, fixed geometry matters but only for tight routing spaces. If the second fails outright, the diagnostic study stands alone as the contribution.