Diversity Regularization in Mixture of Experts Models

115 concepts83 papers1 proposal

The core philosophy behind Mixture-of-Experts (MoE) architectures is intuitive: divide and conquer. Rather than training a single monolithic network to master every pattern across an entire dataset, an MoE framework routes different inputs to a bank of specialized sub-networks, or "experts." In theory, this allows individual components to develop deep, localized expertise, maximizing the model's overall capacity and efficiency.

In practice, however, deep networks are notoriously lazy. Left to standard optimization objectives, multiple experts frequently co-adapt and converge on the exact same dominant features. This phenomenon—known as expert collapse, representational collapse, or expert homogenization—leaves the system with massive parameter redundancy rather than specialized niches.

To understand why this happens, consider how these networks are trained. Standard loss functions evaluate the final aggregated output of the mixture. If one expert happens to discover a strong, easily accessible feature early in training, the gating network will naturally favor it. To minimize loss quickly, the remaining experts are incentivized to mimic this successful strategy rather than risk exploring alternative, subtler patterns. Even standard load-balancing losses, which explicitly force the gating network to distribute tokens evenly across all available experts, only solve the problem of routing volume. They ensure that every expert receives data, but they do not guarantee that the experts actually learn different things. As Hongcan Guo et al. observe in Advancing Expert Specialization for Better MoE, aggressive load balancing can ironically worsen expert homogenization by forcing different sub-networks to process identical data distributions without any functional differentiation.

To break this cycle of redundancy, a growing body of research focuses on diversity regularization—introducing explicit auxiliary losses that mathematically penalize similarity between experts, forcing them to occupy distinct functional roles.


Output-Space Diversity Constraints

The most direct way to prevent expert collapse is to look at what the experts predict. If two sub-networks consistently output identical probability distributions for the same input, they are functionally redundant. By penalizing similarity in the output space, we can push their decision boundaries apart.

In InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generation, Jinqi Xiao et al. address this by implementing an output-space orthogonality loss. By penalizing the dot product of different experts' outputs, the constraint acts as a repulsive force, ensuring that each module develops distinct functional roles capable of handling diverse, multi-conditional tasks without interfering with one another. Similarly, Chenzhengyi Liu et al. utilize a Jensen-Shannon (JS) divergence penalty in DimonGen: Diversified Generative Commonsense Reasoning for Explaining Concept Relationships. By forcing the probability distributions of different modules to diverge, the regularization prevents the system from defaulting to a single, most-likely relationship, preserving a rich variety of specialized reasoning paths.

However, completely pushing output distributions apart can sometimes be too restrictive. For many tasks, multiple experts should agree on the primary, correct label. To accommodate this, Lianbo Zhang et al. introduce a clever masking strategy in Learning a Mixture of Granularity-Specific Experts for Fine-Grained Categorization. Their diversity constraint masks the highest-probability class before calculating the similarity between expert outputs.

Expert A Output: [ 0.85 (Class 1),  0.10 (Class 2),  0.05 (Class 3) ]
Expert B Output: [ 0.82 (Class 1),  0.03 (Class 2),  0.15 (Class 3) ]
                     │
                     ▼  (Mask Primary Class 1)
Expert A Tail:   [  0.10 (Class 2),  0.05 (Class 3) ]
Expert B Tail:   [  0.03 (Class 2),  0.15 (Class 3) ]
                     │
                     ▼  (Penalize Similarity Here)

By focusing the penalty strictly on the tail of the distribution, the experts are allowed to agree on the primary classification while being forced to develop entirely unique internal representations of the relationships between secondary categories.


Representational and Hidden-State Decorrelation

While output-space constraints prevent experts from making identical final predictions, they do not stop experts from learning redundant internal features along the way. A deeper approach targets the internal representations directly, ensuring that the latent spaces spanned by the experts remain distinct.

In SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse, Giang Do et al. tackle representational collapse by directly measuring and minimizing the similarity between the hidden states of different experts using Centered Kernel Alignment (CKA). They argue that directly penalizing statistical similarity in the hidden layers provides an explicit geometric guarantee of functional specialization that indirect methods, like routing adjustments, simply cannot match.

Other geometric approaches enforce this decorrelation directly on the latent distributions or parameter spaces. For instance, Zhihao Wu et al. apply a Maximum Mean Discrepancy (MMD) loss to the latent outputs of different experts in Where Graph Meets Heterogeneity: Multi-View Collaborative Graph Experts, forcing them to map data into non-overlapping feature spaces. Even when experts are carved out of a shared parameter pool—as seen in the upcycling framework proposed by Shengzhuang Chen et al. in Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts—an orthogonality penalty can be enforced directly on the learnable parameter masks. This ensures that individual sub-networks extract distinct features while maintaining enough underlying structural overlap for collaborative synergy.


Information-Theoretic and Geometric Repulsion

Beyond simple distance and alignment metrics, some frameworks leverage information theory or point processes to maximize the collective coverage of the expert bank. The goal here is to treat the experts as a cohesive team where every member must contribute entirely non-redundant information.

Jinyu Cai addresses this in Mixture of Experts as Representation Learner for Deep Multi-View Clustering by introducing a pair-wise kernel regularizer in a Reproducing Kernel Hilbert Space (RKHS) designed to minimize the mutual information between different experts. Without this explicit minimization, learners naturally drift toward redundant functions; with it, they are forced to capture complementary, orthogonal facets of the data.

Alternatively, we can view the experts as a set of points that we want to scatter as widely as possible across a functional landscape. In ST-ExpertNet: A Deep Expert Framework for Traffic Prediction, Hongjun Wang et al. employ a Determinant Point Process (DPP) as a diversity-promoting loss function.

$$\mathcal{L}_{\text{diversity}} = -\log \det(L)$$

Here, $L$ represents a similarity matrix between the outputs of the different experts. Because the determinant of a matrix corresponds to the volume of the parallelepiped spanned by its vectors, maximizing the determinant mathematically punishes volume overlap. The DPP acts as a powerful repulsive force, driving individual experts to occupy entirely unique functional niches.


Why Diversity Regularization Works

Why go through the trouble of adding these complex auxiliary losses? The intuition boils down to robustness, parameter efficiency, and coverage.

When a model encounters highly skewed or long-tailed data distributions, unregularized multi-expert systems collapse into a single majority-biased predictor because the dominant classes overwhelm the early training phase. As Fenyu Hu et al. demonstrate in GraphDIVE: Graph Classification by Mixture of Diverse Experts, enforcing diversity-promoting regularization preserves unique decision boundaries for minority classes, preventing the system from ignoring rare but critical data points.

Furthermore, maintaining distinct expert identities serves as a powerful shield against distribution shifts. Jinyong Jeong et al. show in Multi-Expert Distributionally Robust Optimization for Out-of-Distribution Generalization that preventing multiple classifiers from blending into a single average model ensures that the final ensemble retains the diverse perspectives necessary to handle unexpected out-of-distribution shifts during inference. By explicitly penalizing redundancy, diversity regularization transforms a collection of redundant sub-networks into a highly cooperative, specialized team.

Key Foundational Work

Learning a Mixture of Granularity-Specific Experts for Fine-Grained Categorization (2019) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 84 papers
  2. Version 1 · 2026-10-04 · 90 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_expert_diversity_regularization_594b18f5,
  title = {expert diversity regularization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-diversity-regularization/rdr_594b18f5724649d8b7700e33e13aac1d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_expert_diversity_regularization_594b18f5_v2,
  title = {expert diversity regularization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expert-diversity-regularization/rdr_594b18f5724649d8b7700e33e13aac1d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake