Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery

Shengze Xu, Maolin Deng, Tieyong ZengarXiv 2026 · arXiv

TL;DR

This paper provides a rigorous mathematical framework for understanding "routing collapse" in Mixture-of-Experts (MoE) models—where the model ignores certain experts—and characterizes the exact dynamics of how these suppressed experts "recover" during training. By decomposing the velocity of routing changes, the authors show that recovery times follow predictable logarithmic or power-law scales depending on the geometry of the gradient signal.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) models are the secret sauce behind scaling massive LLMs, allowing them to have trillions of parameters while only activating a small fraction for any given input. However, they suffer from a notorious problem: routing collapse.

During training, the "router" (the part of the model that chooses which experts to use) often develops a strong preference for a few experts, leaving the others "suppressed" with near-zero probability. Because a suppressed expert receives no data, it stops improving, creating a self-reinforcing cycle. While researchers use "load-balancing losses" to force even distribution, we don't fully understand the underlying physics of how these probabilities move near the boundaries. This paper asks: Why does concentration persist even when experts are trainable, and what exactly determines how long it takes for a "dead" expert to come back to life?

Methods

The authors treat softmax routing as a gradient flow problem. They analyze the system using several novel mathematical tools:

  • Normal-Velocity Decomposition: This is the paper's centerpiece. It breaks down the "speed" at which a suppressed expert’s probability increases into two components:
  • Complementary-expert covariance: How much the other experts vary in their performance.
  • Relative gradient advantage: How much "better" the suppressed expert is compared to the current average.
  • Stationary-Loss Barrier: A geometric method to identify which "face" of the probability simplex (i.e., which subset of experts) the model will naturally gravitate toward when experts are fixed.
  • Discrete Log-Odds Identity: A formula that translates discrete neural network updates (like those from AdamW) into changes in the "log-odds" of an expert being selected. This allows the theory to be tested on real-world Transformer architectures, not just abstract math.

Datasets

The researchers validated their theories across a hierarchy of environments:

  • Constructed Vector Fields: Synthetic simulations used to verify the precise logarithmic and power-law recovery time predictions.
  • AG News: A standard text classification dataset used to train a 6-layer Transformer-MoE model.
  • Additional Benchmarks: The appendices mention experiments on Digits (MNIST-like), UCI HAR (Human Activity Recognition), and a large Four-domain corpus (ArXiv, Gutenberg, etc.) to show that routing differences persist across different scales and tasks.

Key Insights and Ideas

  • Boundary Degeneracy: The authors highlight that the softmax function has a "flat" geometry near zero. Even if an expert is significantly better than its peers, the router responds incredibly slowly when that expert's probability is near the boundary. This is a structural property of softmax, not just a training fluke.
  • The "Aha!" of Recovery Scales: Recovery isn't just "fast" or "slow." The authors prove it follows specific regimes. If the recovery signal is strong and steady, recovery is logarithmic ($T \approx \log(1/\epsilon)$). If the signal vanishes as the expert gets closer to the boundary, recovery becomes a much slower power-law ($T \approx \epsilon^{-\alpha}$).
  • Covariance as a Lifeline: Interestingly, an expert can recover even if it doesn't have a clear "advantage" over others, provided there is high covariance among the other experts. Diversity in the active experts can actually help a suppressed expert get back into the game.
  • Persistence vs. Interior Limits: They show that while trainable experts eventually lead to an "interior" solution (using all experts), the "slow adaptation" of experts means that for most of a finite training run, the model behaves as if it has collapsed.

Key Results

  • Theoretical Accuracy: In synthetic tests, numerical "hitting times" (the time to reach a probability threshold) matched the theoretical predictions with a relative error below $4 \times 10^{-7}$.
  • Transformer Interventions: In real Transformer-MoE training, the authors performed a "gradient intervention"—specifically removing a class-conditional contrast in the router's gradient. This intervention delayed confirmed recovery by 44 to 59 optimizer steps and significantly reduced the accumulated log-odds for suppressed experts.
  • Capacity Constraints: The findings held true both with and without capacity constraints (top-k routing), proving that these dynamics are fundamental to the softmax router itself, regardless of the specific dispatch policy.

Limitations and Future Work

  • Square-Loss Focus: Much of the rigorous theoretical proof relies on a common-target square-loss model. While the experimental results on Transformers (using cross-entropy) suggest the theory generalizes, the mathematical "guarantees" are currently tightest for square loss.
  • Fixed Expert Assumption: The "stationary-loss barrier" analysis assumes experts are fixed, which is a simplification of the "joint" evolution where experts and routers change simultaneously.
  • Future Directions: The authors suggest that future work should look at how these recovery coefficients evolve naturally in massive-scale systems and whether specific "gradient directions" can be engineered to force faster recovery of under-utilized experts.

Summary updated

Abstract

Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others. We study how this concentration forms, why it persists, and how suppressed probabilities recover. For fixed experts, a stationary-loss barrier identifies the face supporting the optimal routed output. In a common-target square-loss model with freely trainable expert outputs, slow adaptation tracks fixed-expert dynamics on finite horizons, while every fixed positive adaptation rate yields an interior routing limit. An exact normal-velocity decomposition separates complementary-expert covariance from relative gradient advantage. Under positive two-sided power bounds along recovery paths, uniform over initial masses, the resulting coefficient's order of vanishing gives logarithmic or power-law recovery times. Integrations of constructed fields reproduce these orders. An exact discrete log-odds identity relates neural updates to threshold crossing. In paired Transformer-MoE training, removing a class-conditional router-gradient contrast delays confirmed recovery and reduces accumulated log-odds on fixed tokens, both with and without capacity constraints.
BibTeX
@misc{xu2026softmax,
  author = {Shengze Xu and Maolin Deng and Tieyong Zeng},
  title = {{Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery}},
  year = {2026},
  eprint = {2610.39442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39442},
}
Figure