Crowding by Similarity: Routing Mixture-of-Experts Models by Effective Expert Diversity

A mixture-of-experts model (MoE) is a network that holds many interchangeable feed-forward blocks, the "experts," and uses a small learned module, the router, to send each token to only one or a few of them. Such models fail in two ways that pull against each other. Left alone, the router funnels most tokens to a few experts and the rest starve and die. Forced to spread tokens evenly, the experts see the same mix of data and learn the same function, up to 99% similar in Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer. Population genetics has a long-standing account of this double failure. A herd's census size says little about its genetic health. What matters is the effective population size, which falls both when a few individuals do most of the breeding and when the breeders are close relatives. Commercial duck lines are numerous yet show the smallest effective size and an inbreeding coefficient of 0.26 ± 0.15, while smaller local breeds keep more diversity (Whole-genome resequencing reveals genetic differentiation and selection signatures among wild, local and commercial duck populations). Current MoE fixes treat one axis at a time. Load balancers equalize who gets tokens. Similarity penalties push experts apart regardless of how much they are used.

We propose to measure and control a single quantity that couples the two axes: the effective number of functionally distinct experts, weighted by how much each is used. The mechanism we borrow is negative frequency-dependent selection, the process that keeps rare variants alive in natural populations. In our router, the cost of sending a token to expert $i$ is the total load on all experts that compute something similar to $i$, not just the load on $i$ itself. Tokens are therefore pushed out of crowded functional niches rather than merely out of busy experts. The two-sentence version: load balancing counts heads, but collapse is about how many distinct heads there are. We price routing by functional crowding, so that experts can specialize unevenly without collapsing into copies of one another.

Core Claim

Let $p_i$ be the long-run fraction of tokens routed to expert $i$, tracked as an exponential moving average. Let $K_{ij}\in[0,1]$ be the similarity of experts $i$ and $j$, measured on a shared probe set of tokens, with $K_{ii}=1$. We define the effective expert number as

$$D = \frac{1}{p^\top K p}.$$

This is the order-2 similarity-sensitive diversity index from ecology. It equals $E$ when all $E$ experts are equally loaded and fully distinct. It equals 1 when experts are identical or when one expert takes everything. A dead expert contributes nothing however distinct it is.

The routing controller adds a bias $b_i$ to each router logit and updates it, without gradients, as

$$b_i \leftarrow b_i - \eta\left[(Kp)_i - p^\top K p\right],$$

where $\eta$ is a step size. The term $(Kp)_i$ is the load in expert $i$'s functional neighbourhood. Setting $K=I$ recovers a pure load-price controller, close to the congestion cost of $\phi$-Balancing for Mixture-of-Experts Training. That paper's contribution was to move balancing from the mini-batch to the data population, using a moving-average price per expert. Its price for expert $i$ depends only on $p_i$. Ours depends on who else is doing the same job.

The claim has three parts. First, $D$ tracked early in training predicts final quality and specialization better than load-only statistics, namely the Gini coefficient and exponentiated routing entropy. Second, the similarity-weighted controller beats load-only pricing at matched compute. Third, it also beats the simpler sum of a load balancer and a separate similarity loss.

The controller never adds a penalty on expert weights. Diversity is meant to arise because experts receive different data. This is exactly what Advancing Expert Specialization for Better MoE argues batch balancing prevents.

Research Program

The first part is diagnostic. On MoE language models trained under Switch-style batch balancing, $\phi$-balancing, and no balancing, we will log $p$ and $K$ at every checkpoint. We will then ask whether $D$ separates runs that routing entropy calls identical.

We will borrow one further lens from livestock genomics. There, short runs of homozygosity mark old inbreeding and long runs mark recent bottlenecks (Genetic Diversity and Selection Signatures Within Diannan Small-Ear Pigs Revealed by Next-Generation Sequencing). Our counterpart is to date when the off-diagonal mass of $K$ appears. Homogenization imprinted in the first few thousand steps would call for early intervention. Homogenization that appears during fine-tuning would call for an intervention at adaptation time. SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse found collapse is worst in early layers, so we will estimate $K$ and run the controller per layer.

The second part is the controller itself. Every $f$ steps, a fixed random probe of 256 tokens is passed through all experts in a layer. $K$ is taken as linear centered kernel alignment (CKA, a standard score for how alike two sets of features are) between expert outputs, smoothed by a moving average. This costs about $256E$ extra expert evaluations per update and avoids the full pairwise comparison over training tokens that forced SimSMoE to skip checks. The biology literature also supplies a caution. Contemporary effective size of the orchid Holcoglossum pumilum ranged from about 12 to over 560 depending on data filtering (Genomic Evidence of Low Contemporary Effective Population Size and Southern Genetic Reservoirs in an Island Endemic Epiphytic Orchid of Taiwan). We will therefore report $D$ and controller performance under three similarity estimators: output CKA, output cosine, and cosine of flattened weights after a random projection. We will also vary probe size. A result that holds under only one estimator will be treated as fragile.

Baselines

The naive reference is a token-choice top-2 MoE with the Switch-style auxiliary balancing loss. This is the default that the file's sources identify as causing homogenization. It shows whether any of the methods here matter at all at our scale.

The strongest existing competitor is $\phi$-balancing with negative-entropy potential ($\phi$-Balancing for Mixture-of-Experts Training). It already solves the population-versus-batch problem with a gradient-free moving-average price, so it is the method our controller must beat head to head. Its reported gains on Moonlight-16B-A3B are Math500 accuracy from 64.8% to 66.2% and HumanEval from 73.2% to 75.9%. We cannot train at that scale. We will therefore reimplement it in our own models, with the same data, token budget, and sweep over step size and potential as our controller.

The second competitor adds a similarity penalty to $\phi$-balancing. We will use either SimSMoE's CKA loss or the orthogonality and variance losses of Advancing Expert Specialization for Better MoE. This tests whether coupling load and similarity in one price matters, or whether two independent fixes suffice. We will also include the orthogonal-update optimizer of Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer at small expert counts, where it is known to work. That paper reports average GLUE gains of 0.62 for BERT and 0.9 for RoBERTa.

The key ablation is our controller with $K=I$. It is identical in code, compute, and tuning, so any gap isolates the similarity weighting. A second ablation shuffles $K$ across expert indices each update. It keeps the similarity statistics but breaks their link to identity, which rules out a gain from extra noise alone.

Every method gets the same number of tuning trials, 12 per method. We will report mean and spread over three seeds. Beating a baseline means lower validation perplexity at equal tokens outside seed spread, together with a gap in downstream scores that is in the same direction.

Why not just…?

Why not just use exponentiated routing entropy as the "effective number of experts"? That quantity is the $K=I$ case of $D$. By construction it is blind to homogenization, which is the failure the orthogonal-optimizer paper documented under balanced load. The diagnostic study tests this directly. If entropy-based counts predict final perplexity and specialization as well as $D$ across our runs, the diagnostic half of this proposal is unnecessary and we will say so.

Why not just add a CKA or orthogonality loss on top of $\phi$-balancing? This is the real threat. Penalizing similarity equally for all pairs spends effort on idle experts and pushes busy experts apart even where they should share features. Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement found this kind of over-regularization hurts on simple data. The orthogonal optimizer loses room to move as expert count grows. Still, a well-tuned sum could match us. The additive baseline is included, with equal tuning budget, precisely to settle this. If it ties, the contribution shrinks to a measurement tool.

Why not just use hard balanced assignment or expert-choice routing? BASE Layers: Simplifying Training of Large, Sparse Models guarantees equal token counts but must revert to greedy routing at autoregressive inference. Expert-choice routing, as studied in Routers in Vision Mixture of Experts: An Empirical Study, drops tokens under fixed capacity. Both equalize head counts rather than distinctness, so they would score well on load-only diversity and potentially poorly on $D$. Our controller is a per-expert bias, so inference uses the same greedy router it trained with. We will log $D$ for a BASE run to check whether equal assignment still homogenizes.

Proposed Experiments

We will pretrain decoder-only MoE models of roughly 100M to 500M active parameters on C4, as in the $\phi$-balancing setup, and on FineWeb (The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale). Expert counts will be 16, 64, and 128 with top-2 routing.

Primary metrics are validation perplexity on C4, FineWeb, and WikiText-103. We will also track $D$ per layer, and two specialization checks. The first is domain purity: the mutual information between expert choice and source domain, on code, math, and prose slices. The second is the number of experts receiving under 0.1% of tokens.

For a direct link to prior numbers, we will train a character-level model on enwik8. There, SimSMoE reports 1.08 bits per character against 1.11 for its baseline at 135M parameters. Downstream, we will fine-tune on GLUE, SQuAD 1.1, and BANKING77, where SimSMoE reports a 4.8% accuracy gain. At our largest scale we will also report zero-shot MMLU against the FineWeb reference of 33.6% at 38B tokens. For Math500 and HumanEval, no reported result found on Lacuna at our model scale, so we rely on rerun baselines only.

Three checks decide the proposal. First, the controller must beat $\phi$-balancing and the $K=I$ ablation in perplexity by more than seed spread. The gap should widen with expert count, since that is where orthogonal-subspace methods fail and where functional crowding has the most room to occur. Second, it must match or beat the additive $\phi$-balancing-plus-similarity-loss baseline at equal tuning. Third, $D$ measured at 10% of training must rank-correlate with final perplexity across all runs more strongly than Gini or routing entropy.

If the first check passes but the second fails, we will report the result as a diagnostic contribution only. If $D$ loses the prediction test, the population-genetic framing adds nothing beyond load statistics, and we will report that negative result.