Rethinking Sparse Mixture of Experts from a Unified Perspective

Giang Do, Hung Le, Truyen Tran

TL;DR

The paper introduces Unified Sparse Mixture of Experts (USMoE), a novel routing framework that reinterprets expert selection as a global linear programming problem. By abandoning rigid, fixed per-token or per-expert budgets in favor of a global allocation strategy, USMoE dynamically routes compute to where it matters most. This unified approach delivers substantial performance gains, slashes inference costs, and drastically improves model robustness against noise across both language and vision tasks.

What Problem Does This Paper Address?

Sparse Mixture of Experts (SMoE) architectures are highly popular because they allow large language and vision models to scale up their parameter counts without a corresponding explosion in computational cost. They do this by activating only a small subset of specialized sub-networks ("experts") for any given input token.

However, existing SMoE models are held back by a fundamental routing dichotomy:

  1. Token Choice (TC): Each token gets a fixed budget of experts. This approach struggles with noisy or irrelevant tokens (wasting compute on them) and often suffers from unbalanced expert loading.
  2. Expert Choice (EC): Each expert gets a fixed budget of tokens. This ensures balanced loading but introduces a severe "token dropping" problem where critical tokens might not be selected by any expert, leading to information loss. It also causes information leakage in autoregressive models.

Both approaches rely on rigid, localized budgets and are highly susceptible to representation collapse—a failure mode where the router gets lazy and repeatedly selects the same small subset of experts, destroying model capacity.

Methods

The authors step back and reframe expert selection through the lens of linear programming. Instead of making isolated routing decisions per token or per expert, they optimize expert assignment globally across the entire sequence.

USMoE introduces two primary components:

  • Unified Score Function: It computes a routing score by taking a linear combination ($\alpha \cdot S_e + \beta \cdot S_t$) of both row-wise (Token Choice) and column-wise (Expert Choice) softmax distributions. This blends token-centric and expert-centric preferences.
  • Unified Mechanism: Instead of running independent top-k operations for each token, USMoE flattens the entire token-expert score matrix and performs a global top-k selection.

This allows the model to dynamically allocate more experts to highly complex, informative tokens while completely skipping or reducing compute for noisy, redundant, or padding tokens.

Illustration of Token Choice, Expert Choice, and the proposed globally optimized USMoE selection mechanism.

Datasets

The authors rigorously evaluated USMoE across a wide variety of tasks, domains, and scales:

  • LLM Reasoning: ARC-Challenge, ARC-Easy, BoolQ, OpenBookQA, PIQA, and WinoGrande (evaluated on Qwen3-30B architectures).
  • Text Embeddings (MTEB subset): 15 tasks covering Classification, Clustering, Pair Classification, Reranking, Retrieval, Semantic Textual Similarity (STS), and Summarization (evaluated on OLMoE, Qwen1.5-MoE, and DeepSeekMoE).
  • Language Modeling Pre-training: Enwik8, Text8, WikiText-103, and One Billion Words (lm1b).
  • Vision Classification: CIFAR-10, CIFAR-100, STL-10, SVHN, and ImageNet-1K (evaluated on ViT-MoE).

Key Insights and Ideas

  • Global Competition Filters Noise: When tokens compete globally for expert slots, clean and highly contextual tokens naturally outcompete noisy or corrupted ones. This global ranking acts as an inherent filtering mechanism, making the model remarkably resilient to input perturbations like typos or adversarial attacks.
  • Fractional Top-K Budgets: Because routing is handled globally, USMoE naturally supports fractional average expert allocations. The authors found they could match the performance of a rigid $K=8$ Token Choice baseline using a global budget equivalent to an average of just $K=3.5$ experts per token, unlocking massive computational savings.
  • Mathematically Defeating Collapse: By combining two distinct normalization mechanisms (row-wise and column-wise), USMoE enriches the mathematical structure (specifically increasing the effective rank of the routing Jacobian). This prevents experts from converging onto identical representations during training.

Key Results

  • Robustness Under Attack: In zero-shot evaluations on Qwen3-30B-A3B-Instruct under corrupted environments (15% random Gaussian noise), USMoE outperformed the original model by an average of 13.5%.
  • Halved Sensitivity: Despite being subjected to noise attacks, USMoE consistently demonstrated router sensitivity scores that were roughly half those of competing Token Choice and Expert Choice baselines.
  • Pre-training Efficiency: When training a 20M parameter Transformer-XL from scratch, USMoE achieved a 14% reduction in FLOPs while outperforming standard Token Choice routing.
  • Perfect Long-Context Retrieval: On the grueling Needle-in-a-Haystack task at a 16.5K token context length, USMoE achieved 100% retrieval accuracy, compared to 50% for Token Choice and a dismal 10% for Expert Choice.
  • Vision Superiority: When integrated into Vision Transformers (ViT-MoE), USMoE delivered the highest mean accuracy across all image classification datasets with significantly lower variance than SoftMoE or Token Choice.

Limitations and Future Work

  • Scale Limitations: Due to compute constraints, the authors limited their largest evaluations to medium-scale models (up to Qwen3-30B). Testing USMoE on ultra-large models exceeding 100B parameters remains an open task.
  • Baselines: The paper does not compare USMoE against the newest crop of massive, reasoning-optimized MoE models (such as DeepSeek-R1 style architectures).
  • Need for Custom Kernels: While globally flattening and sorting matrices is mathematically elegant and reduces FLOPs, practical wall-clock speedups at massive batch sizes and extreme context lengths will require highly optimized, custom hardware kernels for global token-expert selection.

Summary updated

Abstract

Sparse Mixture of Experts (SMoE) models scale the capacity of models while maintaining constant computational overhead. SMoE methods fall into two categories: Token Choice, which routes each token to a fixed number of experts, and Expert Choice, which assigns a fixed number of tokens to each expert. However, the use of fixed budgets for tokens or experts causes both approaches to select irrelevant token-expert pairs or overlook critical assignments, which degrades overall performance. To fill that gap, we rethink SMoE from a unified perspective through the lens of linear programming, which provides a general formulation for SMoE models. Furthermore, we introduce Unified Sparse Mixture of Experts (USMoE), a novel framework comprising a unified mechanism and a unified score to overcome these limitations. We provide both theoretical justification and empirical evidence demonstrating USMoE's effectiveness. Extensive evaluations across diverse data settings (clean and corrupted), multiple domains (including texts and vision tasks), and different learning approaches (training-free and training-based) show that USMoE not only delivers significant performance improvements over existing SMoE methods, but also enables more flexible expert selection budgets, reducing inference costs without compromising model performance. Our implementation is publicly available at https://github.com/giangdip2410/USMoE.
BibTeX
@inproceedings{do2026rethinking,
  author = {Giang Do and Hung Le and Truyen Tran},
  title = {{Rethinking Sparse Mixture of Experts from a Unified Perspective}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=PMYAEsMEm5},
}
Figure