A mixture-of-experts model (MoE) is a network that holds many parallel sub-networks, the experts, and uses a small gating model, the router, to decide which experts process each input token. Today's routers assign every token to its best-scoring experts, even when the token sits in a region that several experts cover equally. Quantum detection theory has studied exactly this situation for decades. When two quantum states overlap, no measurement can tell them apart perfectly. One well-understood answer, unambiguous state discrimination, never guesses. It reports "state $i$" only when the evidence rules out every other state, and it sends all overlapping probability mass to an explicit "inconclusive" outcome.
We propose a router built on that rule. Each token is split into two parts. The part that lies uniquely in one expert's territory goes to that specialist. The part shared across experts goes to a shared generalist expert, and its weight is computed rather than hand-set. The idea has three consequences. Experts are allowed to overlap, so we do not need the hard orthogonality that recent anti-collapse methods impose. When experts collapse toward each other, the shared weight rises, so collapse becomes a measurable and self-penalizing quantity. Finally, the "shared expert" pattern that interpretability work finds in large MoEs gets a principled routing rule.
Core Claim
Representation collapse is a known failure in MoEs: experts become near-copies, or token representations are squeezed into a low-dimensional space. Two lines of work respond by pushing experts apart. On the Representation Collapse of Sparse Mixture of Experts shows that dot-product routing gradients pull tokens into the span of the expert embeddings. Its fix, X-MoE, routes on a normalized low-dimensional sphere with a learned temperature. Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts (MOORE) forces expert outputs to be exactly orthonormal with a Gram-Schmidt layer. Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer (OMoE) projects expert updates onto the orthogonal complement of the other experts. That last paper also supplies the constraining evidence. Experts in standard MoE language models reach up to 99% similarity, yet enforcing orthogonality hurts on small datasets such as CoLA and MRPC, and the method weakens as the expert count grows.
Our claim is that the right condition is weaker than orthogonality. It is the condition for unambiguous discrimination, which requires each expert to have some direction not spanned by the others; for single vectors this means linear independence. The overlap that remains should be routed explicitly to a shared expert. Unambiguous discrimination between mixed quantum states proves that this support condition is necessary and sufficient for error-free conclusive outcomes. It also bounds the inconclusive probability from below by the pairwise overlaps (fidelities) of the states. That paper supplies the mechanism. Decoding Knowledge Attribution in Mixture-of-Experts motivates the design: it finds that large MoEs already divide labor between always-on shared experts and selectively routed specialists, but there the split is an architectural choice rather than a per-token computation.
How the Router Works
Each token $x$ is projected to a small routing space of dimension $d_r$ (16 to 64, following X-MoE) and normalized to $\hat{x}$. Each routed expert $i \in \{1,\dots,E\}$ owns a learned direction $w_i$; a rank-$r$ subspace is an option for the mixed-state version. From the $E \times E$ Gram matrix we compute dual vectors $\tilde{w}_i$ with $\langle \tilde{w}_i, w_j \rangle = 0$ for $i \neq j$. These define measurement operators $E_i = c_i\, \tilde{w}_i \tilde{w}_i^{\top}$ and an inconclusive operator:
$$E_0 = I - \sum_{i=1}^{E} c_i\, \tilde{w}_i \tilde{w}_i^{\top} \succeq 0,$$
The scalars $c_i \ge 0$ are chosen as large as possible while keeping $E_0$ positive semidefinite. A uniform choice, $c_i = 1/\lambda_{\max}(\sum_i \tilde{w}_i\tilde{w}_i^\top)$, is differentiable and costs one small eigenvalue computation. The optimal $c_i$ solve an $E$-variable semidefinite program, using the formulation of Optimal quantum detectors for unambiguous detection of mixed states; we refresh this solution every few hundred steps.
The token's routing weights are then:
$$p_i(x) = c_i\,(\tilde{w}_i^{\top}\hat{x})^2, \qquad p_0(x) = 1 - \sum_{i=1}^{E} p_i(x).$$
The layer output is $y = p_0(x)\,S(x) + \sum_{i \in \text{top-}k} p_i(x)\,F_i(x)$. Here $S$ is the shared expert and $F_i$ are the routed experts. Sparsity comes from top-$k$ selection over $p_1,\dots,p_E$.
Two properties follow directly. First, a token lying entirely in the span of the other experts' directions gives expert $i$ exactly zero weight. Specialists therefore see only the part of the data they uniquely own. A softmax router gives no such guarantee.
Second, the inconclusive mass grows with expert overlap; in the pure-state bound from the Feng et al. paper it is at least $\sqrt{\tfrac{E}{E-1}\sum_{i\neq j}\eta_i\eta_j |\langle w_i,w_j\rangle|^2}$, where $\eta_i$ is expert $i$'s routing frequency. If experts drift toward each other, $p_0$ approaches one and the routed experts starve. A standard load-balancing loss over the routed experts then pushes back against collapse, without forcing orthogonality. One knob controls how much overlap is tolerated: an optional penalty that targets a desired average $p_0$.
Research Program
The first stage builds the router as a drop-in replacement for top-$k$ softmax gating in an encoder MoE and in the MOORE multi-task RL encoder. It then checks three behaviors. Expert similarity should stay below the collapse regime. Routing should fluctuate less from step to step than with softmax gating, since the zero-weight property removes ties between overlapping experts. And $p_0$ should concentrate on tokens that are genuinely ambiguous, such as function words, cross-domain tokens, and shared manipulation primitives.
The second stage tests the claim that bounded non-orthogonality beats hard orthogonality where tasks share structure. The focus is low-resource fine-tuning and many-expert settings, which are exactly where OMoE reports degradation.
The third stage, kept small, returns the idea to its physical origin. In the simulated quantum MoE of QMoE: A Quantum Mixture of Experts Framework for Scalable Quantum Neural Networks, the router uses a computational-basis measurement. We replace it with the same unambiguous measurement, implemented by an ancilla dilation. The inconclusive outcome then controls a shared quantum expert. We also add depolarizing noise to probe the effect described in Strictly contractive quantum channels and physically realizable quantum computers: noise shrinks the distance between states, so the inconclusive rate should rise smoothly under noise rather than routing collapsing abruptly.
Baselines
The naive reference is a dense model with matched active parameters. Alongside it we run a standard top-$k$ softmax MoE with a load-balancing loss and no shared expert. These show whether routing helps at all at our scale.
The strongest classical competitor on the collapse question is X-MoE from On the Representation Collapse of Sparse Mixture of Experts. It uses the same low-dimensional normalized routing space as our router, so the only difference is the gating rule. That paper reports a masked-LM validation perplexity of 18.72 against 19.02 for the sparse-MoE baseline on CCNet/Wikipedia, and an XTREME average of 65.3 against 63.8. We cannot reproduce that compute. We will re-implement X-MoE at our smaller scale with an identical data stream, token budget, expert count, active parameters, and an equal hyperparameter search (same number of trials over learning rate and temperature, or $p_0$ target for ours). Beating it means lower perplexity than our X-MoE reproduction across three seeds by more than the seed spread, with lower expert similarity. The published numbers serve only to sanity-check that our reproduction lands in a comparable relative gap over plain sparse MoE.
The strongest competitor on the orthogonality question differs by domain. In multi-task RL it is MOORE, which reports 88.7% success on MetaWorld MT10 and 72.9% on MT50 in Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts. We use MOORE's released setting and swap only its Gram-Schmidt layer for our router, with the same number of experts and environment steps. Beating it means matching MT10 and improving MT50 beyond seed variance. MT50 has more shared structure among tasks, which is where hard orthogonality should hurt most. In language fine-tuning it is OMoE from Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer, which reports average GLUE gains of +0.62 (BERT) and +0.9 (RoBERTa) over AdamW. We run it with the same backbones and tuning budget. Success means matching its GLUE average while not degrading on CoLA and MRPC, and doing so without its roughly 1.38x memory overhead.
The baseline that most directly threatens the idea is a shared-expert MoE with a confidence gate. It keeps the softmax router, adds an always-on or confidence-gated shared expert, and sets the shared weight to $1 - \max_i \text{softmax}_i$ or to normalized routing entropy. This captures the "send ambiguous tokens to a generalist" intuition with no measurement theory.
Three ablations isolate our key ingredient, the dual-basis decomposition. The first drops $E_0$ and renormalizes the $p_i$. The second keeps $E_0$ but routes its mass to a randomly chosen routed expert. The third uses the minimum-error "pretty good measurement" of Quantum-inspired algorithm for direct multi-class classification as the router. That router is quantum-inspired but always guesses, so it tests whether the inconclusive outcome, rather than quantum-style geometry, does the work.
Why not just…?
Why not just add an always-on shared expert with a confidence threshold? This is the real threat, and it is why the confidence-gated baseline exists. A confidence gate is one scalar per token computed from softmax scores. It does not stop a specialist from receiving tokens that lie in another specialist's region. It also gives no signal that rises when experts collapse, because two near-identical experts can still produce a confident softmax. Our router differs on both counts, and we will measure each directly: per-expert input purity (the fraction of a specialist's inputs whose dominant component lies in its unique direction), and the correlation between $p_0$ and expert similarity over training. If the confidence gate matches us on perplexity, purity, and collapse, the decomposition adds nothing. We would then report the result as negative and keep only the diagnostic use of $p_0$.
Why not just orthogonalize the experts, as MOORE does? Orthogonality removes all shared directions. The attribution analysis in Decoding Knowledge Attribution in Mixture-of-Experts suggests that real tasks share foundational features. OMoE's own results show that forcing apart experts trained on small datasets destabilizes routing. Our router keeps the shared directions and assigns them to the generalist. Whether that matters is settled by the MT50 comparison and the CoLA/MRPC fine-tuning runs.
Why not just use a cosine-similarity penalty between experts? MOORE reports that soft similarity penalties fail to prevent collapse in multi-task RL. Improving Multi-Task Generalization via Regularizing Spurious Correlation needed decorrelation losses combined with causal routing graphs and invariant-risk training before decorrelation became stable. We include a tuned cosine-penalty MoE in the GLUE runs, given the same trial budget. Our expectation is that it either under-separates experts or hurts the task at the penalty weights where collapse is avoided.
Why not just use X-MoE's sphere routing? X-MoE fixes the gradient pull toward expert centroids, but it still forces every token into a top-$k$ assignment. Our router shares X-MoE's routing space, so the head-to-head comparison isolates the gating rule alone.
Proposed Experiments
The language-modeling study pretrains encoder MoEs of roughly 100 to 300 million total parameters, with 8, 16, and 32 experts, on a Wikipedia-scale corpus. Each run tracks validation perplexity, mean pairwise expert-output cosine similarity, routing fluctuation (the fraction of tokens whose top expert changes between checkpoints), and the $p_0$ distribution. The sweep over expert counts tests whether our advantage over orthogonal methods grows as more experts make orthogonality harder to satisfy, the point where OMoE weakens. We then fine-tune on GLUE with the CoLA and MRPC splits reported separately.
The reinforcement-learning study runs MetaWorld MT10 and MT50 with five seeds against MOORE.
The quantum check reruns QMoE's 4-class MNIST setting at 8x8 resolution, where QMoE reports 62.92% accuracy against 51.47% for a monolithic quantum network. It compares computational-basis routing with unambiguous routing under increasing depolarizing noise.
The decisive checks are three. First, the dual-basis router must beat the confidence-gated shared-expert baseline on perplexity and input purity at 32 experts; if not, the core claim fails. Second, removing $E_0$ must recover softmax-like collapse, with expert similarity rising toward the reported 99% regime. Third, on MT50 and on CoLA/MRPC, bounded overlap must outperform hard orthogonality. If the second check fails, the anti-collapse argument is wrong even if accuracy improves. If only the third fails, the contribution narrows to a principled shared-expert gate rather than a replacement for orthogonality.
References
- On the Representation Collapse of Sparse Mixture of Experts
- Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
- Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
- Unambiguous discrimination between mixed quantum states
- Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
- Optimal quantum detectors for unambiguous detection of mixed states
- QMoE: A Quantum Mixture of Experts Framework for Scalable Quantum Neural Networks
- Strictly contractive quantum channels and physically realizable quantum computers
- Quantum-inspired algorithm for direct multi-class classification
- Improving Multi-Task Generalization via Regularizing Spurious Correlation