In a sparse mixture-of-experts (MoE) network, each token is sent by a small learned gate (the "router") to a few specialist sub-networks ("experts"). The layer's output is the gate-weighted average of those experts' outputs. That average is a straight-line interpolation between points in activation space. The generative-modeling literature has shown for years that straight-line interpolation between points on a curved data manifold lands in regions the next module has never seen. We propose that this is not a side issue for MoE. We think it is one of the forces behind expert collapse, the well-known failure in which experts either starve or converge to near-identical functions. If two experts have truly specialized, their outputs point in different directions, and averaging them produces an off-manifold vector that the downstream layers handle badly. Gradient descent can remove that penalty in two ways: make routing so lopsided that mixing rarely happens, or make the experts agree so that mixing becomes harmless. Both are forms of collapse.
The concrete opportunity is to change how expert outputs are combined, not how tokens are routed. Every existing fix for collapse that we know of works on the router: its geometry, its probability distribution, or its balancing loss. The other half of the layer is untouched. If combining outputs in a manifold-respecting way lets experts specialize further at the same load balance and quality, the result gives both a new explanation for homogenization and a cheap architectural change.
Core Claim
Write the MoE layer output as $y = \sum_{i \in S} g_i E_i(x)$, where $x$ is the token's hidden state, $S$ is the set of selected experts, $g_i$ are the normalized gate weights, and $E_i(x)$ is expert $i$'s output. By the triangle inequality,
$$\|y\| \le \sum_{i \in S} g_i \|E_i(x)\|,$$
with equality only when all selected expert outputs point the same way. The gap between the two sides grows as experts become less aligned, which is exactly as they become more specialized. So linear mixing of specialized experts contracts outputs radially inward toward the origin. More generally, it moves them off the distribution of single-expert outputs.
This is the same signature that Off-Manifold Collapse in Guided Protein Language Models found when steering dense protein models: activations contract inward, and a diagonal Mahalanobis distance flags the degenerate states. It is also the chordal-interpolation problem that motivates spherical and geodesic interpolation in Nonlinear Image Interpolation using Manifold Learning and Geodesic Calculus on Implicitly Defined Latent Manifolds.
Our claim has two testable parts. The first is diagnostic. In trained MoE language models, tokens with high gate entropy (tokens near routing boundaries) produce mixed outputs that are measurably off-manifold relative to single-expert outputs, and this off-manifold score predicts elevated per-token loss. The second is causal. Replacing linear mixing with a combination that stays on-manifold reduces the pressure toward homogenization. The result should be lower expert similarity at equal load balance and equal or better perplexity, and the gain should be concentrated on high-entropy tokens.
Relation to Prior Work
Router-side fixes motivate this work but leave the combination step untouched. On the Representation Collapse of Sparse Mixture of Experts shows through a Jacobian argument that dot-product routing pulls hidden states toward expert centroids. Its fix, X-MoE, moves routing onto a low-dimensional hypersphere. EMoE routes by alignment with a learned orthonormal principal-component basis. Routing Manifold Alignment (RoMA) regularizes a post-trained router so that its choices match those of semantically similar, correctly answered neighbors in an external embedding space. All three change which experts are chosen. None asks whether averaging the chosen experts' outputs is itself the harmful operation.
Distributional and diversity methods also constrain the design. Dirichlet-Prior Shaping pushes router probabilities toward the corners of the simplex. Breaking the Echo Chamber (MP-MoE) penalizes redundant expert co-activation. The orthogonal optimizer of Diversifying the Mixture-of-Experts Representation forces experts apart directly. Under our hypothesis, forcing diversity without fixing mixing should hurt boundary tokens, so these methods are both competitors and a natural test.
Weight-space merging is a separate matter. Expert Merging with Nash Bargaining shows that linearly averaging expert parameters washes out specialization. Our concern is averaging expert activations for each token at inference time, which is a different object.
The method side borrows from off-manifold work. We reuse two ideas. The first is the learned projection operator from Geodesic Calculus on Implicitly Defined Latent Manifolds, a network trained by denoising to map points back onto the data support. The second is the rank-one normal-versus-tangent decomposition of Manifold-Optimal Guidance, which penalizes off-manifold drift while allowing movement along the manifold. We deliberately do not compute pull-back Riemannian metrics. The file notes that these become expensive in high dimensions and have only been demonstrated up to about 84 dimensions, far below MoE hidden sizes.
Research Program
Measurement. For each MoE layer we fit a reference distribution to single-expert outputs, meaning outputs from tokens whose top gate weight exceeds 0.9. We fit a diagonal Gaussian, following the protein-model paper, and also a small learned denoising projector. The off-manifold score of any output is its Mahalanobis distance under that reference, together with its radial contraction ratio $\rho = \|y\| / \sum_i g_i \|E_i(x)\|$. We compute these scores on the mixed outputs and also on the post-layer hidden state, since the residual connection may partly absorb the drift.
Interventions. We test three increasingly expressive replacements for linear mixing, each of which keeps the gate weights and routing unchanged. The first is norm-restoring mixing. It rescales $y$ so that $\|y\| = \sum_i g_i \|E_i(x)\|$. This keeps the direction of the linear mix but undoes radial contraction, and it is the chordal-to-spherical correction. The second is tangent-projected mixing. It removes the component of $y - E_{\text{top}}(x)$ that lies along a learned per-layer normal direction, in the spirit of the rank-one metric in Manifold-Optimal Guidance. The third is learned projection. Here $y$ passes through a low-rank residual projector $\Pi$. This projector is trained jointly, with a denoising loss on noised confident single-expert outputs and a stop-gradient into the experts, so that it cannot absorb the task.
The ordering matters. If the cheapest correction captures the whole effect, the mechanism is radial contraction and nothing more, and we will report it that way.
Baselines
The naive reference is a standard token-choice top-$k$ MoE with linear mixing and the usual auxiliary load-balancing loss, as used in OLMoE. We train it at the same size, on the same data, and with the same token budget as every variant, and we sweep the balancing coefficient over the same grid. Beating it means lower validation perplexity together with lower pairwise expert-output similarity at matched load imbalance.
The strongest existing competitor on diversity at pre-training time is MP-MoE. Its authors report pre-training on a 100B-token subset of FineWeb-EDU and outperforming the OLMoE baseline by 1–3 percentage points across downstream tasks, at about 3% training overhead. The file gives no absolute baseline numbers for it. We reimplement it in our setup with its co-occurrence penalty tuned over the same budget of runs we give our method. Beating it means matching or exceeding its downstream gain with at most comparable overhead, and doing better specifically on high-gate-entropy tokens.
For the routing-geometry fix we run X-MoE's hyperspherical router. That paper reports a masked language modeling validation perplexity of 18.72 against 19.02 for a standard sparse MoE on CCNet and Wikipedia. We use it to check whether routing geometry and combination geometry help independently, by crossing X-MoE routing with our mixing variants.
Two further baselines isolate the key ingredient. The first is top-1 routing, where no mixing occurs at all. It tells us whether the benefit comes from better mixing or simply from not mixing. The second is a post-mixture LayerNorm-only control. It changes output scale without using any manifold reference, and it separates "restore norm" from "restore typicality."
RoMA is a post-training method. Its reported 69.0% MMLU on OLMoE comes from router-only fine-tuning, so we treat it as an orthogonal add-on rather than a pre-training competitor. We apply it on top of our best variant only if the pre-training results warrant it.
Why not just…?
Why not just use top-1 routing, so no interpolation ever happens? This is the most serious alternative. If our hypothesis is correct, top-1 should also show less homogenization. Top-1 gives up the soft capacity and gradient signal that multi-expert routing provides, and OLMoE's design settles on top-8 with fine-grained experts. The experiments settle it directly: if top-1 matches our best mixing variant in perplexity at equal active compute, the claim reduces to "avoid mixing," and we will say so.
Why not just force experts apart with an orthogonality or co-activation penalty? Diversifying the Mixture-of-Experts Representation and MP-MoE do this and report gains. Our hypothesis predicts a specific cost: forced diversity with linear mixing should raise the off-manifold score and the loss on boundary tokens, even if average loss improves. We measure exactly this per-token breakdown. If diversity penalties show no boundary-token cost, our mechanism is not the bottleneck.
Why not just add a normalization after the MoE layer? Transformers already normalize around sublayers, so radial contraction might already be neutralized. That is why the pre-norm and LayerNorm-only controls are included, and why we measure off-manifold scores on the post-residual hidden state. If existing normalization already removes the loss penalty on high-entropy tokens in OLMoE, the diagnostic fails and the interventions are unnecessary.
Proposed Experiments
Diagnosis on OLMoE. The first experiment needs no training. On the released OLMoE-1B-7B model, which uses 64 experts per layer with top-8 routing, we compute off-manifold scores and contraction ratios on FineWeb-EDU validation text. We then regress per-token loss on the off-manifold score while controlling for gate entropy and token frequency. OLMoE reports that routing saturates within about the first 1% of training. Using open intermediate checkpoints where they are available, we test whether expert-output similarity rises during that window alongside a falling mixture off-manifold score. That pattern would be consistent with homogenization acting as a repair.
Controlled pre-training. In small MoE language models (roughly 1B total parameters, top-2 and top-8) trained on a fixed FineWeb-EDU token budget, we cross the three mixing variants with the balancing-loss coefficient grid. We report four things: validation perplexity overall and by gate-entropy decile; pairwise expert similarity, using the pairwise embedding similarity metric of Mixture of Experts with Soft Nearest Neighbor Loss; load imbalance; and wall-clock overhead. Downstream evaluation uses MMLU and GSM8K. At this scale we expect small absolute numbers, so we compare only within our own runs. The file's reported GSM8K result for ProbMoE (50.19% against 45.94% for conventional routing on an OLMoE backbone) comes from a much larger model and is not a like-for-like target.
Decisive checks. The hypothesis survives only if all three of the following hold. First, off-manifold score predicts per-token loss in OLMoE beyond gate entropy alone. Second, at matched load imbalance, at least one manifold-respecting mixing variant yields lower expert similarity together with equal or better perplexity than both linear mixing and top-1, with the gain concentrated in the highest gate-entropy deciles. Third, with our mixing in place, the balancing-loss coefficient can be lowered further than with linear mixing before quality degrades. That would show combination geometry, and not only the router, shaping the balance–specialization trade-off. If only norm-restoring mixing is needed, the contribution narrows to an explanation of collapse as radial contraction plus a one-line fix, which is still a useful and falsifiable result.
References
- Off-Manifold Collapse in Guided Protein Language Models
- Nonlinear Image Interpolation using Manifold Learning
- Geodesic Calculus on Implicitly Defined Latent Manifolds
- On the Representation Collapse of Sparse Mixture of Experts
- EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts
- Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs
- Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
- Breaking the Echo Chamber: A Dynamic Ensemble Pruning Perspective on MoE
- Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
- Expert Merging in Sparse Mixture of Experts with Nash Bargaining
- Manifold-Optimal Guidance: A Unified Riemannian Control View of Diffusion Guidance
- OLMoE: Open Mixture-of-Experts Language Models
- Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement