Is Expert Collapse a Memory-Capacity Failure? A Testable Capacity Law for Mixture-of-Experts Routers

A mixture-of-experts model splits one large feed-forward layer into many smaller "experts." A small scoring network, the router, sends each token to a few of them. When this is trained naively, two failures appear. Most tokens pile onto a handful of experts while the rest go unused, or, when the model is forced to spread tokens evenly, the experts become near-copies of one another. Meanwhile, research on associative memories (networks that store patterns and retrieve the closest one when given a cue) has worked out exactly when retrieval breaks. Past a critical number of stored patterns, which depends on the dimension of the space and how correlated the patterns are, cues stop landing on one clean memory and instead land on blurred mixtures of several. A router is structurally the same object. It computes a softmax over dot products between the token and one stored vector per expert, which is the one-step retrieval rule of a modern Hopfield network.

This note proposes to test whether that structural match predicts something. The hypothesis is that expert collapse sets in when the router is asked to separate more experts than the effective dimensionality of its inputs can support. It further proposes that the onset follows the exponential capacity law derived for dense associative memories, applied to the effective dimension of the token representations the router sees rather than their nominal width. If the law holds, it tells practitioners how many experts a given router can use before extra experts are wasted, and it identifies the lever that matters: the geometry of the router's inputs, not the spacing of the expert vectors. If the law fails, that is also informative, because it would show the Hopfield analogy for routing to be decorative.

Core Claim

The router's softmax over expert scores is the update rule $\xi \mapsto X\,\mathrm{softmax}(\beta X^\top \xi)$ that Hopfield Networks is All You Need derived from a log-sum-exp energy. In the routing case, the stored patterns $X$ are the $E$ expert vectors, the query $\xi$ is the token's router input, and $\beta$ is the router's inverse temperature. That paper, and the statistical-physics analysis in Exponential Capacity of Dense Associative Memories, show that clean retrieval holds up to roughly $\exp(\alpha N)$ patterns in dimension $N$ and then fails abruptly at a critical load. Effects of Feature Correlations on Associative Memory Capacity shows that correlated patterns shrink that threshold.

Transformer hidden states are strongly correlated and concentrate in few directions. The relevant $N$ for a router is therefore not the hidden width. It is the effective dimension of the router input distribution, which we define as the participation ratio of its covariance eigenvalues $\lambda_i$:

$$d_{\text{eff}} = \frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}.$$

The prediction is that the number of experts at which routing turns "metastable" (tokens spread their weight across many experts rather than committing to a few), and at which dead or duplicated experts start to accumulate, obeys

$$\log E_{\text{crit}} \approx \alpha_c(\beta)\, d_{\text{eff}},$$

where $\alpha_c$ is a load constant that depends on temperature but not on model width. The form comes directly from the $P=\exp(\alpha N)$ threshold with $N$ replaced by $d_{\text{eff}}$.

The empirical signature is concrete. Collapse metrics plotted against $\log E / d_{\text{eff}}$ should fall onto one curve across router widths, and they should not fall onto one curve when plotted against $E$ alone or against tokens per expert.

Nominal widths of 4096 would put this threshold far out of reach. The claim is only interesting because $d_{\text{eff}}$ can be small and because fine-grained designs push $E$ into the thousands. Little By Little (MoRAM) is one such design, with thousands of rank-1 experts and no separate router. Establishing where real models sit relative to the threshold is part of the work, and we state it as an open empirical question rather than an assumption.

Relation to Prior Work

The closest work on the memory side is Provably Optimal Memory Capacity for Modern Hopfield Models. It frames stored memories as spherical codes (points on a sphere placed as far apart as possible), proves capacity $M^\star \asymp c^{D_\Phi}$, and introduces U-Hop+, which spreads the memories apart by gradient descent. On MNIST it raised pure single-memory retrieval from about 3.5% to 100%. It never touches routing. The obvious transfer is to apply U-Hop+ to expert vectors, and our law predicts this transfer is incomplete. Spreading the expert vectors does not help if the tokens themselves occupy a low-dimensional slab, because queries confined to few directions cannot resolve many well-spaced keys. The proposal therefore treats key separation as an ablation, not as the method.

On the mixture-of-experts side, three lines of work fix collapse with training rules. BASE Layers solves an exact balanced assignment per batch. $\phi$-Balancing balances load over a running population estimate using convex duality. OMoE projects expert updates onto orthogonal subspaces. OMoE is the one that constrains our design most directly: it reports that its gains shrink as the number of experts grows, because the free subspace runs out. That is exactly what a dimension-bounded capacity law would predict, and our experiments can test whether the degradation point matches $d_{\text{eff}}$.

Statistical Advantages of Perturbing Cosine Router shows that cosine routing with a small constant added to the norm converges faster. Because its router already lives on a sphere, it is the natural base for our method. Mixture of Experts with Soft Nearest Neighbor Loss motivates the input-geometry view. It separates representations before routing and cuts expert similarity (SIM, the mean pairwise similarity of expert outputs) on CIFAR100 from 0.20 to 0.10. However, it needs a separate pretraining stage, over-regularizes easy data, and offers no quantitative prediction.

None of these works predicts the expert count at which collapse begins. The research file itself identifies this as open: no formulation exists of the maximum number of specialized experts a router of dimension $D$ can support.

Research Program

The first component is measurement. We instrument router inputs in small language models and in open mixture-of-experts checkpoints to estimate $d_{\text{eff}}$ per layer, and to estimate the "metastable size" of each token. Following Hu et al.'s MNIST measurement, metastable size is the number of experts whose pre-top-$k$ softmax weight exceeds a fixed fraction of the maximum. Alongside these we track standard collapse indicators: dead-expert fraction, expert specialization entropy, and pairwise expert similarity (SIM).

The second component is a controlled phase-diagram study. Decoder language models with 100M to 400M parameters are trained on FineWeb-Edu from FineWeb, which $L^3$ also used. The router receives a learned projection of the hidden state with width $d_r$ between 8 and 256. We sweep the number of experts $E$ from 8 to 4096 in fine-grained configurations that keep active parameters fixed. We also sweep router temperature. Varying $d_r$ gives direct control over $d_{\text{eff}}$. A separate arm varies $d_{\text{eff}}$ without changing $d_r$, by whitening or by deliberately anisotropic projections. This separates "effective" from "nominal" dimension.

The third component is a calibration check against theory. We fit $\alpha_c$ from synthetic retrieval experiments on Gaussian patterns with a matched covariance spectrum. Then we ask whether the same $\alpha_c$ predicts onset in the trained models without refitting.

The fourth component is an intervention derived from the law: raise the router's effective dimension instead of fighting the router with penalties. Router inputs pass through a running whitening transform followed by a random-feature map that lifts them to a higher $D_\Phi$. This follows Dense Associative Memory Through the Lens of Random Features and the kernelized models of Hu et al. Temperature is set from the fitted $\alpha_c$, so that the target $E$ sits below the critical load. We call this a capacity-aware router. It costs one extra matrix multiply per layer, is identical at training and inference time, and needs no batch-level coordination.

Baselines

The naive reference is a standard softmax top-$k$ router with a Switch-style auxiliary balance loss, tuned over three loss weights. This is the configuration most open models ship. If our law cannot explain where this baseline collapses, the whole proposal fails.

The strongest competitor on router design is the perturbed cosine router of Statistical Advantages of Perturbing Cosine Router. Its reported enwik8 bits-per-character is 1.197, against 1.213 for a vanilla cosine router, and its reported DomainBed average accuracy is 68.37%. We reproduce it inside our codebase on enwik8 and on FineWeb-Edu, with identical data, token budgets, and a matched sweep over its perturbation constant. Because our absolute numbers will depend on model size, we report the reproduced gap rather than comparing against the published 1.197 directly.

The strongest competitor on load balancing is $\phi$-Balancing. It reports improvements at 16B scale, from 64.8% to 66.2% on Math500 and from 73.2% to 75.9% on HumanEval. That scale is beyond this study, so we reimplement it at our scale and give it the same tuning budget as our method, set as a fixed number of runs per method. OMoE is included in the large-$E$ arm, where its reported degradation with expert count is the relevant comparison.

Two ablations isolate the key ingredient. The first applies U-Hop+-style separation of expert vectors while leaving inputs unchanged. The second whitens inputs without the random-feature lift. To beat the baselines, our method must hold dead-expert fraction and SIM flat out to at least twice the $E$ at which the best baseline begins to degrade, at equal or lower perplexity or bits-per-character.

Why not just…?

Why not just add an orthogonality or spreading penalty on the expert vectors? This is the direct transfer of U-Hop+, and it is cheap. The law predicts it will help only while $d_{\text{eff}}$ exceeds what the target $E$ requires. Beyond that point, well-spaced keys cannot be resolved by queries confined to few directions. The key-separation ablation tests this directly. If it matches the full method at large $E$, the input-dimension claim is wrong, and we would report that.

Why not just call this data starvation? With more experts, each expert sees fewer tokens and fewer gradients. That alone could produce dead experts with no geometry involved, and it is the most serious rival explanation. We separate the two by scaling batch size with $E$ so that tokens per expert stay constant, and by varying $d_{\text{eff}}$ at fixed $E$. Under the starvation account, onset tracks tokens per expert and ignores $d_{\text{eff}}$. Under ours, the reverse holds.

Why not just use $\phi$-balancing or BASE-style assignment? These guarantee even load. OMoE and the SNNL paper show that even load can coexist with near-duplicate experts, and BASE layers must fall back to greedy routing at inference. Our experiments measure SIM and metastable size under $\phi$-balancing across the $E$ sweep. If homogenization under balancing still appears at the predicted $\log E / d_{\text{eff}}$, balancing addresses a symptom rather than the limit.

Why not just put a LayerNorm or whitening before the router? This is a real threat to the intervention: the whitening-only ablation may capture most of the benefit. If it does, we accept that outcome. The contribution then shifts to the capacity law, which says when whitening is needed and when it is not enough, while the method reduces to a one-line change.

Decisive Checks

The central test is curve collapse. Across at least four router widths and the full $E$ sweep, dead-expert fraction and median metastable size should fall onto a single curve against $\log E / d_{\text{eff}}$, with a residual clearly smaller than when plotted against $E$ or tokens per expert. The threshold should also be sharp, consistent with the all-or-nothing transition in Exponential Capacity of Dense Associative Memories.

A second check targets the effective-dimension claim specifically. At fixed nominal $d_r$, artificially flattening or steepening the input spectrum should move the onset in the predicted direction.

A third check probes MoRAM's growth claim. Rank-1 atoms are accumulated on the TRACE sequence, where MoRAM reports 1.37% backward transfer. We measure retrieval purity as atoms are added and ask whether degradation begins where the law says. The law gives MoRAM's linear self-activation a finite ceiling, which is testable.

Finally, the capacity-aware router must match or beat the reproduced perturbed-cosine and $\phi$-balancing baselines on FineWeb-Edu validation perplexity, Wikitext-103, and enwik8 bits-per-character, while extending the collapse-free range of $E$.

A negative result on the first check, with no curve collapse under effective dimension, would falsify the proposal cleanly.