Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement

Abien Fred Agarap, Arnulfo P. AzcarragaarXiv 2026 · DOI

TL;DR

This paper tackles the "expert collapse" problem in Mixture-of-Experts (MoE) models, where experts become redundant and the gating network struggles to route effectively. They propose a novel approach using a feature extractor pre-trained with Soft Nearest Neighbor Loss (SNNL) to disentangle class representations, leading to more specialized experts and improved performance on complex datasets.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) models are powerful ensemble techniques where a "gating network" learns to route input data to specialized "expert networks." This allows for efficient learning by dividing the problem space. However, a common issue, "expert collapse," arises when the input features have overlapping class boundaries. This forces the gating network into rigid routing strategies, and experts may learn similar, redundant representations, hindering the model's overall performance and generalization. The paper aims to resolve this by ensuring experts learn truly distinct, specialized roles.

Methods

The core innovation is the introduction of a feature extractor network that is optimized before the MoE architecture. This feature extractor is trained using Soft Nearest Neighbor Loss (SNNL). SNNL is a technique that encourages class-similar data points to be closer together in the learned representation space, while pushing class-dissimilar points further apart. By applying SNNL to the feature extractor, the input data is "primed" with disentangled class information. This pre-conditioned representation is then fed to the MoE's gating and expert networks. The authors also introduce metrics like Expert Specialization Entropy and Pairwise Embedding Similarity to quantify expert diversity and routing flexibility.

Datasets

The proposed method was evaluated on four benchmark image classification datasets:

  • MNIST
  • FashionMNIST
  • CIFAR10
  • CIFAR100

Key Insights and Ideas

  • SNNL as a "pre-conditioning" step: Instead of directly applying SNNL to the entire MoE, using it to optimize a dedicated feature extractor is key. This disentangles the input space before it reaches the experts, making the gating network's job easier.
  • Resolving "structural expert collapse": The paper argues that SNNL doesn't just lead to harder routing, but fundamentally changes the internal representations learned by the experts, making them more orthogonal and less redundant.
  • Trade-off on simple datasets: For very simple datasets like MNIST, the SNNL pre-conditioning can lead to "over-regularization," slightly degrading performance compared to a baseline MoE. This suggests the benefit is most pronounced on complex, entangled data.
  • Quantifying specialization: The proposed metrics (Entropy and Similarity) provide a more nuanced understanding of MoE behavior beyond just accuracy, revealing how experts are collaborating (or not).

Key Results

  • Improved Accuracy on Complex Datasets: The SNNL-augmented MoE model achieved statistically significant improvements in classification accuracy on FashionMNIST, CIFAR10, and CIFAR100 compared to a baseline MoE. For example, CIFAR100 accuracy increased from 35.75% to 36.74%.
  • Reduced Expert Redundancy: Pairwise Embedding Similarity (SIM) values were significantly lower for the SNNL-enhanced models on complex datasets (e.g., CIFAR100 SIM dropped from 0.20 to 0.10), indicating experts learned more distinct features.
  • Increased Routing Entropy: On FashionMNIST and CIFAR100, routing entropy increased, suggesting a more flexible and collaborative routing strategy by the gating network.
  • No significant accuracy gain on MNIST: The baseline model outperformed the SNNL-enhanced model on MNIST, highlighting the potential for over-regularization on simple datasets.

Limitations and Future Work

  • Over-regularization on simple datasets: The authors acknowledge that SNNL can be detrimental on very simple, easily separable datasets like MNIST.
  • Dynamic Parameter Tuning: Future work could explore dynamically adjusting the SNNL weighting factor ($\alpha$) and temperature parameter ($T$) during training to better optimize the transition from clustering to classification.
  • Architectural Scaling: The current experiments used lightweight CNNs. Scaling the approach to more complex architectures like Vision Transformers (ViTs) and larger datasets (e.g., ImageNet) is suggested.
  • Comparative Representation Learning: Benchmarking against other modern contrastive learning techniques is needed to isolate the specific benefits of SNNL for MoE gating.

Summary updated

BibTeX
@misc{agarap2026mixture,
  author = {Abien Fred Agarap and Arnulfo P. Azcarraga},
  title = {{Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement}},
  year = {2026},
  eprint = {2603.26734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.26734},
}
Figure