Routers That Track Their Experts: A Representational-Similarity Account of the MoE Diversity Trade-off

A mixture-of-experts (MoE) model splits each feed-forward layer into many sub-networks ("experts") and uses a small learned model, the router, to send each token to only a few of them. These models often fail in one of two ways, both called expert collapse. Either a few experts receive almost all tokens, or the experts become near-copies of each other. The fixes that push experts apart, such as orthogonality penalties, similarity penalties, and projected optimizers, help, but they come with a puzzle. One paper reports that making experts too different destabilizes the router. Another reports that more different experts make routing better.

We propose that both reports are right, and that one measurable quantity separates them. That quantity is whether the router's view of which tokens are alike matches the experts' view of which tokens are alike. Representational similarity analysis (RSA), a method from neuroscience, compares two systems by checking whether they rank the same pairs of inputs as similar, without needing their coordinates to line up. It gives a direct way to measure this match. If the match predicts router instability, a cheap loss that keeps the router aligned with the experts should let expert diversification go further without the instability.

Core Claim

The router decides where a token goes from the hidden state before the experts run. The experts, in turn, define after the fact which tokens they treat alike. When a diversity penalty reshapes the experts, the second geometry moves and the first may not follow. Our hypothesis is that router instability under diversification is caused by this growing mismatch, and not by diversity itself.

We test this in two steps. The first is diagnostic: over training, we track the RSA alignment between the router's similarity structure and the experts' similarity structure. We ask whether a fall in alignment comes before routing churn and loss spikes. The second is an intervention: an alignment loss that pulls the router's similarity structure toward the experts' structure. A stop-gradient on the expert side makes the router follow the experts and prevents the experts from collapsing toward the router.

Concretely, on a probe set of $n$ tokens sampled from each batch, let $K_r$ be the $n \times n$ Gram matrix (pairwise inner products) of the router's log-probability vectors. Let $K_e$ be the Gram matrix of the concatenated outputs of all experts on those same tokens, computed densely and with gradients stopped. The added term is

$$L_{align} = 1 - \mathrm{CKA}(K_r, \mathrm{sg}[K_e]),$$

where CKA (centered kernel alignment) is a normalized similarity between two Gram matrices that ignores rotations and overall scale, and $\mathrm{sg}$ denotes stop-gradient.

How this differs from closest work

The closest work treats similarity in three other places:

  • SimSMoE uses CKA between experts on shared tokens to push them apart. It never compares the router to the experts.
  • SIMBAL regularizes router weights toward orthonormality, $\|R^TR - I\|$. This makes the router preserve the geometry of its input, but it contains no signal about what the experts compute.
  • Advancing Expert Specialization for Better MoE adds an output orthogonality loss $L_o$ and a router variance loss $L_v$. It reports an interaction in which orthogonal experts sharpen router gradients.

These papers motivate the proposal, and one constrains it. OMoE documents up to 99% expert similarity under load balancing and reports that excessive divergence destabilizes routing, especially on small datasets such as MRPC and CoLA. That is the contradiction we aim to resolve.

Our reading is that the $L_o + L_v$ combination may succeed because $L_v$ keeps the router responsive to the experts. OMoE's projection step has no such companion. The diagnostic below tests this directly.

Research Program

Diagnostic study. We train small decoder MoEs under each existing anti-collapse method, with no new loss added. At every checkpoint we compute three things on a fixed held-out probe set of a few thousand tokens:

  • Router–expert alignment: CKA between router and expert-bank Gram matrices.
  • Expert redundancy: pairwise expert similarity, both first-order (mean cosine, as in SIMBAL's pairwise expert similarity) and second-order (CKA between experts' dissimilarity matrices).
  • Routing stability: how often the top-$k$ assignment of a probe token changes between consecutive checkpoints, plus training-loss spike frequency and the ratio of maximum to mean expert load.

The central test is temporal. A drop in router–expert alignment should come before rises in routing churn, and the size of the drop across methods and layers should predict how much diversification hurt.

Layers are analyzed separately. SimSMoE finds collapse is worst in early layers. The Expert Strikes Back finds that router specialization emerges early while expert output specialization peaks late. This suggests mismatch should be largest in middle-to-late layers, where expert geometry changes most.

Guarding against confounds. Obstacles to inferring mechanistic similarity using RSA shows that matching similarity matrices can arise from shared surface features, which is called the mimic effect. In language models the obvious surface feature is token identity and frequency. We therefore recompute alignment after regressing out a token-identity dissimilarity matrix (zero for repeated tokens, one otherwise) and a log-frequency matrix. We report only the residual alignment as evidence for the hypothesis. Similarity scores also depend on the probe distribution, which is the modulation effect, so we report the diagnostic on two probe sets: in-distribution web text and the 5,000-noun set of From Associations to Activations.

Choice of similarity measure. ReSi finds that measure choice is domain-dependent and sensitive to preprocessing. It ranks CKA first in language and finds Orthogonal Procrustes the most consistently stable across domains. We use linear CKA as the primary measure and report Procrustes and Spearman correlation between dissimilarity matrices as robustness checks. Preprocessing (centering, no unit-normalization of columns) is fixed in advance.

Intervention. We add $L_{align}$ on top of each diversification method. Each step samples $n = 256$ probe tokens. Running every expert on these tokens adds $256 \times E$ expert forward passes, where $E$ is the number of experts. CKA on $256 \times 256$ Gram matrices is negligible. The total overhead should be small next to the 1.38× memory cost OMoE reports for its projectors, and we will measure it as wall-clock time per step. Following SimSMoE's practice of applying its penalty only intermittently, we also test computing the term every $f$ steps.

Baselines

Naive reference. The do-nothing reference is a standard top-$k$ MoE trained with the batch load-balancing loss alone. All other methods add to this recipe, and it is the regime in which OMoE reports near-total expert redundancy.

Existing methods. We reimplement four existing methods on the same architecture, data, token budget, and seeds:

  • SIMBAL's orthonormal router penalty.
  • The $L_o + L_v$ pair from Advancing Expert Specialization.
  • SimSMoE's CKA-between-experts loss.
  • OMoE's alternating orthogonal optimizer.

Each method's loss weight gets the same tuning budget: five values on a log grid, selected on held-out loss. Our method gets the same five-value grid.

Strongest competitor. We judge the strongest competitor on stability to be $L_o + L_v$. Its router-variance term is the existing mechanism most likely to already provide what our alignment loss provides. Among methods with reported pretraining numbers in our target regime, SIMBAL is the strongest. It reports 45.19% mean accuracy over ARC, HellaSwag, and PIQA against 43.28% for load balancing, a pairwise expert similarity of 0.0028 against 0.0241, and reaching baseline validation loss with 36% fewer tokens (SIMBAL).

On enwik8, SimSMoE reports 1.08 bits per character against a 1.11 baseline at 135M parameters (SimSMoE). We will not compare our numbers directly to these published figures, because our architectures and budgets will differ. We use them to check that our reimplementations reproduce the reported direction and rough size of gains.

What counts as beating them. Beating the competitors means one of two outcomes:

  • Adding $L_{align}$ to the most aggressive diversity setting each method tolerates lets us raise that method's diversity weight further without instability, and this yields lower held-out loss than the method's own best tuned weight.
  • $L_{align}$ closes the low-data degradation on MRPC and CoLA that OMoE reports.

Ablations. Three ablations isolate the key ingredient, which is that the router is aligned to the experts' geometry:

  1. Align the router to the Gram matrix of its own input hidden states instead of the expert bank. This is a learned-target version of what SIMBAL achieves through weight constraints.
  2. Align to only the selected expert's output rather than all experts.
  3. Align to a random fixed Gram matrix.

If ablation 1 matches the full method, the expert-side signal is unnecessary and the hypothesis is wrong.

Why not just…?

Why not just turn down the diversity penalty until routing stops wobbling? That is the default practice, and it is what the tuning grid already does for every baseline. The question is whether the trade-off frontier can be moved, not where to sit on it. If no weight of $L_{align}$ lets a method run at higher diversity with lower loss than its own best weight, the proposal fails on its central claim.

Why not just combine SIMBAL with the orthogonality loss? This is a real threat. An orthonormal router preserves input geometry, and if expert geometry stays close to input geometry, input preservation alone keeps router and experts in sync. We run SIMBAL plus $L_o + L_v$ as an explicit baseline. Ablation 1 is the cleaner test, because it differs from the full method only in whether the target geometry comes from the experts. The diagnostic adds a further check: if expert-bank geometry drifts away from input geometry during diversification, as we expect in late layers, input preservation cannot track it.

Why not just rely on the router variance loss $L_v$, which already sharpens routing? Advancing Expert Specialization shows $L_v$ makes gating more decisive. Decisiveness, however, is not correctness: a confident router can still send tokens to the wrong experts after those experts have moved. The diagnostic separates the two. If $L_o + L_v$ runs show high router–expert alignment and OMoE runs show low alignment, that confirms our explanation of the contradiction. It would also mean $L_v$ is a partial, indirect version of our loss, and we would report it that way.

Why not just use SimSMoE, which already uses CKA? SimSMoE applies CKA to push experts apart from each other. It has no term relating the router to the experts. It can therefore cause exactly the mismatch our loss repairs, which is why we stack the two rather than substitute one for the other.

Proposed Experiments

Pretraining and diagnostic. We pretrain roughly 150M-active-parameter decoder MoEs with 16 to 64 experts and top-2 routing on FineWeb-Edu. We evaluate zero-shot ARC, HellaSwag, and PIQA, matching SIMBAL's evaluation suite, and include enwik8 for comparability with SimSMoE.

The first decisive check is the diagnostic: a lagged correlation, across methods, layers, and seeds, between drops in residual router–expert alignment and later rises in routing churn. If alignment does not lead churn, or the relation vanishes once token identity is regressed out, we stop. The intervention would then rest on no mechanism.

Frontier test. The second check is the frontier test. For each diversification method we sweep its weight with and without $L_{align}$ and plot held-out loss against second-order expert redundancy. Success means the curve with $L_{align}$ dominates, meaning lower loss at equal or lower redundancy, with the full method beating all three ablations.

Fine-tuning test. The third check repeats the OMoE setting: fine-tuning on GLUE with emphasis on MRPC and CoLA, where OMoE's subspace estimates are reported to fail. The no-alignment comparison is our own reimplemented OMoE run on identical splits and seeds. The comparison is against that run, not against the reported +0.62 to +0.9 average GLUE gains of OMoE over AdamW in OMoE.

Overhead. We report wall-clock and memory overhead throughout, against the 10.7s to 11.5s per-iteration change reported for $L_o + L_v$ on DeepSeek V2 Lite.