Introduction
Sparse Mixture-of-Experts (SMoE) architectures have emerged as a powerful paradigm for scaling deep learning models. By activating only a small subset of specialized subnetworks—"experts"—for any given input, SMoEs achieve high capacity with manageable computational costs. However, a critical challenge remains: determining the optimal number of total experts ($K$) and the number of active experts per token ($k$). Traditionally, this is a manual process of trial and error. Existing adaptive methods, such as DynMoE, expand based on token coverage, often leading to functional redundancy where experts become "jacks-of-all-trades" rather than specialized units.
The MASS Framework
To address these limitations, we introduce MASS (Mixture-of-Experts for Adaptive Semantic Specialization). MASS dynamically grows the expert pool based on semantic necessity, ensuring that each expansion step contributes to genuine specialization rather than just capacity.
Adaptive Expert Expansion
MASS monitors experts for "functional saturation" using a principled two-stage mechanism:
- Gradient Surge Detection: Using probabilistic Change Point Detection (CPD), the framework identifies statistically significant surges in an expert's gradient norm. These surges signal that the expert is under pressure to adapt to data it cannot currently represent well.
- Semantic Differentiation: Once a surge is detected, MASS evaluates the alignment between the new gradient and the expert's existing weights. If the alignment is low (high semantic drift), it indicates the expert is being pulled in divergent semantic directions.
When drift is confirmed, the expert is duplicated. To prevent redundancy, the gradient update is decomposed: the original expert receives only the component aligned with its current role, while the new expert inherits the full update to specialize in the emerging semantic role.
Dynamic Top-$p$ Routing
Beyond expanding the total pool $K$, MASS optimizes the active experts $k$ per token. Instead of a fixed $k$, it employs a confidence-based threshold $p$. The router activates the minimum number of experts required to reach a cumulative probability of $p$. This allows the model to allocate more processing power to "hard" or ambiguous tokens while saving computation on "easy" ones.
Experimental Evaluation
We evaluated MASS across synthetic Hidden Markov Models (HMM), the GLUE benchmark for natural language understanding, and the DomainBed benchmark for vision domain generalization.
Key Results and Efficiency
MASS consistently identifies the "elbow" of the performance-cost frontier—the point of diminishing returns where model performance is maximized relative to computational overhead. In synthetic tests, MASS converged to an average of 12.4 experts and a top-$k$ of 3.9, aligning closely with the manually searched optimal configuration.
On the GLUE benchmark, MASS achieved state-of-the-art accuracy while activating only 25-30% of its expert capacity per token (average $k \approx 3$). This represents a significant efficiency gain over DynMoE, which typically activated 70-90% of its experts. Furthermore, MASS demonstrated robust performance across both language and vision domains, proving that semantic-aware expansion is a versatile strategy for scaling MoE architectures.
Conclusion
MASS provides a principled approach to scaling MoEs by treating gradients as a signal for semantic insufficiency. By automatically finding the optimal balance between $K$ and $k$, it removes the need for expensive manual tuning and prevents expert redundancy. Future work will explore the application of MASS to lifelong learning scenarios and the engineering challenges of scaling gradient-based detection to trillion-parameter models.
Summary updated