How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts

Sumin Park, Noseong ParkarXiv 2025 · DOI · arXiv

Introduction

Sparse Mixture-of-Experts (SMoE) architectures have emerged as a powerful paradigm for scaling deep learning models. By activating only a small subset of specialized subnetworks—"experts"—for any given input, SMoEs achieve high capacity with manageable computational costs. However, a critical challenge remains: determining the optimal number of total experts ($K$) and the number of active experts per token ($k$). Traditionally, this is a manual process of trial and error. Existing adaptive methods, such as DynMoE, expand based on token coverage, often leading to functional redundancy where experts become "jacks-of-all-trades" rather than specialized units.

The MASS Framework

To address these limitations, we introduce MASS (Mixture-of-Experts for Adaptive Semantic Specialization). MASS dynamically grows the expert pool based on semantic necessity, ensuring that each expansion step contributes to genuine specialization rather than just capacity.

Adaptive Expert Expansion

MASS monitors experts for "functional saturation" using a principled two-stage mechanism:

  1. Gradient Surge Detection: Using probabilistic Change Point Detection (CPD), the framework identifies statistically significant surges in an expert's gradient norm. These surges signal that the expert is under pressure to adapt to data it cannot currently represent well.
  2. Semantic Differentiation: Once a surge is detected, MASS evaluates the alignment between the new gradient and the expert's existing weights. If the alignment is low (high semantic drift), it indicates the expert is being pulled in divergent semantic directions.
Comparison of expansion mechanisms: DynMoE (left) adds experts based on unassigned tokens, while MASS (right) uses a principled two-stage mechanism targeting semantic insufficiency.

When drift is confirmed, the expert is duplicated. To prevent redundancy, the gradient update is decomposed: the original expert receives only the component aligned with its current role, while the new expert inherits the full update to specialize in the emerging semantic role.

Dynamic Top-$p$ Routing

Beyond expanding the total pool $K$, MASS optimizes the active experts $k$ per token. Instead of a fixed $k$, it employs a confidence-based threshold $p$. The router activates the minimum number of experts required to reach a cumulative probability of $p$. This allows the model to allocate more processing power to "hard" or ambiguous tokens while saving computation on "easy" ones.

Experimental Evaluation

We evaluated MASS across synthetic Hidden Markov Models (HMM), the GLUE benchmark for natural language understanding, and the DomainBed benchmark for vision domain generalization.

Key Results and Efficiency

MASS consistently identifies the "elbow" of the performance-cost frontier—the point of diminishing returns where model performance is maximized relative to computational overhead. In synthetic tests, MASS converged to an average of 12.4 experts and a top-$k$ of 3.9, aligning closely with the manually searched optimal configuration.

MASS performance relative to the optimal frontier. The framework automatically identifies the 'elbow' point where performance gains meet computational efficiency.

On the GLUE benchmark, MASS achieved state-of-the-art accuracy while activating only 25-30% of its expert capacity per token (average $k \approx 3$). This represents a significant efficiency gain over DynMoE, which typically activated 70-90% of its experts. Furthermore, MASS demonstrated robust performance across both language and vision domains, proving that semantic-aware expansion is a versatile strategy for scaling MoE architectures.

Conclusion

MASS provides a principled approach to scaling MoEs by treating gradients as a signal for semantic insufficiency. By automatically finding the optimal balance between $K$ and $k$, it removes the need for expensive manual tuning and prevents expert redundancy. Future work will explore the application of MASS to lifelong learning scenarios and the engineering challenges of scaling gradient-based detection to trillion-parameter models.

Summary updated

Abstract

Finding the optimal configuration of Sparse Mixture-ofExperts (SMoE) that maximizes semantic differentiation among experts is essential for exploiting the full potential of MoE architectures. However, existing SMoE frameworks either heavily rely on hyperparameter tuning or overlook the importance of diversifying semantic roles across experts when adapting the expert pool size. We propose Mixture-of-Experts for Adaptive Semantic Specialization (MASS), a semanticaware MoE framework for adaptive expert expansion and dynamic routing. MASS introduces two key advancements: (i) a gradient-based semantic drift detector that prompts targeted expert expansion when the existing expert pool lacks capacity to capture the full semantic diversity of the data, and (ii) an integration of adaptive routing strategy that dynamically adjusts expert usage based on token-level routing confidence mass. We first demonstrate that MASS reliably converges to the point of optimal balance between cost-performance trade-off with notably improved sematic specialization in a highly controlled synthetic setup. Further empirical results on real-world datasets across language and vision domains show that MASS consistently outperforms a range of strong MoE baselines, demonstrating its domain robustness and enhanced expert specialization.
BibTeX
@misc{park2025how,
  author = {Sumin Park and Noseong Park},
  title = {{How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts}},
  year = {2025},
  eprint = {2512.19765},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.19765},
}
Figure