A mixture-of-experts model splits a network into several smaller sub-networks, the "experts," plus a small model called the router that decides which expert handles each input. These models fail in a familiar way. An expert that happens to receive slightly more data early on improves faster, the router learns to favor it, and the feedback loop ends with a few overworked experts and many idle ones. This failure is called expert collapse. Quantum versions of this design now exist: the router is itself a quantum circuit that places the input in a superposition over expert indices. In these designs the hardware adds a second force on top of the learning feedback. Quantum states decay. In particular, energy relaxation (the "$T_1$" process, where an excited qubit falls back to $|0\rangle$) pushes every routing register toward the all-zeros bitstring, which is the index of one specific expert. Our claim is that on noisy hardware, quantum mixture-of-experts routers suffer a hardware-induced collapse. Relaxation hands one expert a small, persistent head start, and the classical learning feedback loop then amplifies it. The fix is to remove the asymmetry at the physical level, not to fight it with heavier balancing losses that make all the experts the same.
The second research direction enters through the familiar tradeoff in quantum control between speed and coherence. Slow, adiabatic gates avoid leakage but give the environment more time to corrupt the state. Fast gates limit that exposure but excite unwanted transitions. Router errors are not like expert errors, because the training feedback loop compounds them. We therefore predict that the router circuit should sit at a different point on this tradeoff curve than the expert circuits. We propose to measure that curve and to test a cheap mitigation: random relabeling of expert indices from shot to shot, which turns the directional bias into symmetric noise.
Core Claim
QMoE builds a quantum neural network from a trainable routing circuit and parameterized expert circuits that the router triggers through controlled operations. On 8x8 MNIST and Fashion-MNIST it beats a monolithic circuit. However, all of its experiments use ideal state-vector simulation, and it does not analyze load balancing or collapse at all. That paper motivates this proposal: it supplies the architecture and leaves exactly the noisy regime open.
We make three testable claims. First, under amplitude damping, routing collapses toward the expert indexed by low-Hamming-weight bitstrings, and it does so faster and more completely than under pure dephasing of equal strength. Dephasing destroys phase between expert branches but leaves the populations alone, so it acts like the symmetric routing noise that classical work sometimes injects on purpose. Second, the collapse comes from the interaction of hardware and learning: the bias is small at initialization and grows over training. Third, a per-shot random relabeling removes most of the collapse without the homogenization that strong balancing losses cause. In this relabeling, a random bit mask $m$ is applied to the routing register, and expert $j$ is controlled on $|j \oplus m\rangle$ with classical bookkeeping.
The physics behind the third claim is standard Pauli twirling. Averaging amplitude damping over random $X$ masks maps the channel's fixed point, $|0\ldots0\rangle$, to a uniformly random logical expert. The noise is still there, but it no longer has a favorite expert.
Research Program
We first build a noisy QMoE simulator, since no reported result exists. It uses density-matrix simulation of the QMoE pipeline: phase encoding, a routing circuit over $\log_2 E$ index qubits for $E$ experts, controlled parameterized experts, and aggregation. Each gate has a duration $\tau$. Decoherence is modeled with Lindblad channels: amplitude damping set by $T_1$ and pure dephasing set by $T_2$.
The duration $\tau$ trades off against leakage, so we include a leakage term that grows as gates get shorter. Its form follows the result in Pulse Design of Baseband Flux Control for Adiabatic Controlled-Phase Gates, which shows that leakage scales with the Fourier magnitude of the pulse derivative, so shorter pulses with wider bandwidth leak more. We borrow that paper's leakage model, not its pulse-optimization method. The router and the experts get independent durations, $\tau_R$ and $\tau_E$.
We then measure three quantities across training. The first is the load distribution over experts, read out from routing-register populations and summarized by entropy and a Gini coefficient. The second is the number of dead experts, defined as those receiving under 1% of mass. The third is expert similarity, measured as the overlap of the unitaries each expert implements. This last metric detects homogenization, the opposite failure in which over-balanced experts all learn the same thing. Both failure modes are documented classically: BASE Layers reports starvation, while Advancing Expert Specialization for Better MoE and the orthogonal-optimizer work report homogenization. Both constrain any fix we propose.
The mitigation has two parts. The first is index relabeling, as described above. The second is a population-level balancing correction adapted from $\phi$-Balancing. That method replaces noisy mini-batch balance penalties with an entropy-barrier dual price on a moving average of expert loads, solved by mirror descent. We apply this correction to the measured loads. The reason to use it is that shot noise in quantum readout makes per-batch load estimates especially noisy, and population-level balancing was designed for exactly that problem.
Baselines
The naive reference point is QMoE as published, with no balancing, trained under the same noise model. Its noiseless accuracy anchors the scale. QMoE reports 62.92% on four-class MNIST against 51.47% for a monolithic circuit, and 63.81% on four-class Fashion-MNIST against 53.65% (QMoE). We found no reported result on Lacuna for QMoE under decoherence.
The strongest existing method is QMoE with a classical Switch-style auxiliary loss on the measured routing probabilities. This is the obvious engineering fix, and a learned router can in principle pre-rotate its angles to cancel a fixed relaxation bias.
A second strong competitor is QMoE with $\phi$-Balancing applied to the measured loads, without relabeling. That paper reports gains such as 66.2% versus 64.8% on Math500 in a classical model, so it is a credible balancer in its own domain. It has not been tested in the quantum setting.
All methods share the same simulator, noise parameters, number of shots per step, optimizer, and total number of circuit evaluations. Each method's balancing coefficient gets an equal grid of hyperparameter trials. Two ablations isolate the key ingredient: relabeling alone, and relabeling plus $\phi$-Balancing. To count as beating the strongest baseline, the full method must hold load entropy within 5% of its maximum at a lower expert-similarity score, and match or exceed the baseline's accuracy at the same $T_1$. Accuracy gains bought through homogenization do not count.
Why not just…?
Why not just add a load-balancing loss and let the router learn to compensate? This is the most serious threat. If the relaxation bias is small and constant, gradient descent could cancel it. The classical evidence says strong batch-level losses push experts toward identical behavior ($\phi$-Balancing; Advancing Expert Specialization). But that evidence comes from large language models, not small quantum circuits. Our comparison of the auxiliary loss against relabeling, across a sweep of $T_1$, settles the question directly. If the loss matches relabeling on both load entropy and expert similarity, we will report that the hardware bias is learnable away and drop the relabeling claim.
Why not just use a classical router that dispatches to quantum experts? A classical router never decoheres, so it cannot suffer hardware-induced collapse. It does give up the coherent superposition over experts that QMoE argues for, and it still faces ordinary learning-driven collapse. We include it as a reference arm. If it matches noisy quantum routing on accuracy, that is itself an important negative finding about quantum routing on near-term hardware.
Why not just make the router circuit shorter or faster? Shorter circuits reduce relaxation, but faster gates leak, and fewer gates limit what the router can express. The $\tau_R$ sweep measures this tradeoff explicitly. The interesting outcome is whether the best router speed differs from the best expert speed, which the compounding of router errors through feedback predicts.
Why not just freeze routing early, as StableMoE does? StableMoE distills and freezes routing after roughly 10% of training to stop assignments from fluctuating. A frozen quantum router still decays toward $|0\ldots0\rangle$ at inference time, so freezing does not address a bias that originates in hardware. We test a frozen-router arm to confirm this.
Proposed Experiments
The tasks are four-class 8x8 MNIST and Fashion-MNIST following QMoE, with $E \in \{4, 8\}$ experts. We sweep $T_1$ and $T_2$ independently, comparing pure dephasing against amplitude damping at matched total error. We also sweep the gate durations $\tau_R$ and $\tau_E$ on a grid.
The first decisive check is whether collapse under amplitude damping concentrates on low-Hamming-weight indices while collapse under dephasing does not. Relabeling should erase that index pattern. The second check is whether the load imbalance grows over training rather than being present from the start. Growth over training is the signature of hardware-learning amplification; a static imbalance would mean the bias is purely physical. The third check is whether, on the measured speed-coherence curve, the router's optimal duration is shorter than the experts' optimal duration. If none of these effects appear, then physical decoherence is not a distinct cause of collapse and the classical balancing toolkit is enough. That would also be worth reporting.
References
- QMoE: A Quantum Mixture of Experts Framework for Scalable Quantum Neural Networks
- Pulse Design of Baseband Flux Control for Adiabatic Controlled-Phase Gates in Superconducting Circuits
- BASE Layers: Simplifying Training of Large, Sparse Models
- Advancing Expert Specialization for Better MoE
- Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
- $\phi$-Balancing for Mixture-of-Experts Training
- StableMoE: Stable Routing Strategy for Mixture of Experts