Sparse Mixture of Experts as Unified Competitive Learning

Giang Do, Hung Le, Truyen Tran

Abstract

Sparse Mixture of Experts (SMoE) improves the efficiency of large language model training by directing input tokens to a subset of experts. Despite its success in generation tasks, its generalization ability remains an open question. In this paper, we demonstrate that current SMoEs, which fall into two categories: (1) Token Choice; and (2) Expert Choice, struggle with tasks such as the Massive Text Embedding Benchmark (MTEB). By analyzing their mechanism through the lens of competitive learning, our study finds that the Token Choice approach may overly focus on irrelevant experts, while the Expert Choice approach risks discarding important tokens, potentially affecting performance. Motivated by this analysis, we propose Unified Competitive Learning SMoE (USMoE), a novel and efficient framework designed to improve the performance of existing SMoEs in both scenarios: with and without training. Extensive experiments across various tasks show that USMoE achieves up to a 10% improvement over traditional approaches or reduces computational inference costs by 14% while maintaining strong performance.

Introduction

Sparse Mixture of Experts (SMoE) models have become a cornerstone for scaling massive models like GPT-4 and Mixtral. By activating only a small fraction of parameters for any given input, these models maintain constant computational costs despite having billions of parameters. However, the fundamental mechanism for routing tokens to experts remains problematic.

Current routing strategies generally fall into two categories:

  1. Token Choice (TC): Each token independently selects its preferred experts. This often results in "expert collapse," where a subset of experts is over-utilized while others remain idle. Furthermore, TC is sensitive to noisy or irrelevant tokens.
  2. Expert Choice (EC): Each expert selects the tokens it is best suited to process. While this balances the workload, it frequently "drops" critical tokens that are not selected by any expert. In the context of Large Language Models (LLMs), EC also introduces "information leakage," where the model gains access to future tokens during routing, violating autoregressive training principles.

Proposed Method: Unified Competitive Learning (USMoE)

To address these limitations, we propose Unified Competitive Learning SMoE (USMoE). This framework reinterprets expert selection as a global optimization problem solvable via Linear Programming.

The Unified Mechanism

USMoE moves beyond one-sided selection by calculating a Unified Score, a linear combination of token-centric and expert-centric preferences. The model flattens the similarity matrix between all tokens and experts, selecting the top-N pairs globally. This ensures optimal compatibility matches across the entire sequence.

Sigmoid Routing and Fractional Selection

To eliminate information leakage in EC-style routing, USMoE replaces the standard Softmax function with a Sigmoid function. This prevents tokens from influencing each other's routing scores, making the method safe for autoregressive LLMs. Additionally, USMoE supports Fractional Selection, allowing for non-integer average expert assignments (e.g., 1.5 experts per token). This provides a more granular control over the trade-off between computational cost and model performance.

Key Insights and Visual Analysis

The research highlights several critical advantages of the unified approach:

  • Synergy: Token Choice excels at capturing in-context cues, while Expert Choice is superior at semantic alignment. USMoE leverages both simultaneously.
  • Global Optimality: The "Unified Mechanism" (global Top-K) is mathematically proven to maximize total similarity under a fixed computational budget.
  • Noise Robustness: By taking a global view, USMoE can effectively ignore irrelevant or "junk" tokens by not assigning them to any experts, prioritizing informative input components.
Visual comparison of routing patterns for USMoE, Token Choice (TC), and Expert Choice (EC) across various input images.

Experimental Results

USMoE was evaluated across diverse benchmarks, including the Massive Text Embedding Benchmark (MTEB) for NLP and ImageNet-1K for vision.

  • Performance Gains: USMoE achieved up to a 10% improvement over standard routing on MTEB. Notably, on a summarization task using Qwen1.5-MoE-A2.7B, performance jumped from 13.4% to 40.0%.
  • Efficiency: The framework reduced inference FLOPs by 14% while maintaining competitive accuracy through fractional expert selection.
  • Robustness: In vision tasks, USMoE demonstrated significantly higher resilience to adversarial attacks (e.g., maintaining 46.2% accuracy under PGD attacks compared to 23.4% for TC). Training curves also show much lower perplexity and faster convergence compared to TC and EC, especially in "corrupted" data scenarios.

Conclusion

USMoE provides a robust and efficient alternative to traditional SMoE routing by unifying token and expert preferences. By framing routing as a global optimization problem and introducing Sigmoid-based selection, it achieves superior performance, efficiency, and robustness across both NLP and vision domains. Future work will focus on scaling USMoE to models exceeding 100B parameters and exploring dynamic adjustment of routing parameters during training.

Summary updated

Abstract

Sparse Mixture of Experts (SMoE) improves the efficiency of large language model training by directing input tokens to a subset of experts. Despite its success in generation tasks, its generalization ability remains an open question. In this paper, we demonstrate that current SMoEs, which fall into two categories: (1) Token Choice ;and (2) Expert Choice, struggle with tasks such as the Massive Text Embedding Benchmark (MTEB). By analyzing their mechanism through the lens of competitive learning, our study finds that the Token Choice approach may overly focus on irrelevant experts, while the Expert Choice approach risks discarding important tokens, potentially affecting performance. Motivated by this analysis, we propose Unified Competitive Learning SMoE (USMoE), a novel and efficient framework designed to improve the performance of existing SMoEs in both scenarios: with and without training. Extensive experiments across various tasks show that USMoE achieves up to a 10% improvement over traditional approaches or reduces computational inference costs by 14% while maintaining strong performance.
BibTeX
@inproceedings{do2026sparse,
  author = {Giang Do and Hung Le and Truyen Tran},
  title = {{Sparse Mixture of Experts as Unified Competitive Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=PMYAEsMEm5},
}
Figure