From Sparse to Soft Mixtures of Experts

Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Neil Houlsby

Introduction

Scaling Transformers typically involves increasing parameter counts, which leads to a linear increase in computational cost. Sparse Mixture of Experts (MoE) architectures were introduced to decouple model capacity from the compute budget by routing input tokens to only a subset of "expert" layers. However, traditional sparse MoEs face significant challenges, including training instability due to non-differentiable discrete routing, token dropping when expert buffers overflow, and expert imbalance. Furthermore, operations like top-k sorting are often inefficient on modern hardware accelerators like TPUs and GPUs.

The Soft MoE Approach

To address these limitations, the authors introduce Soft MoE, a fully differentiable architecture that replaces discrete routing with a soft assignment mechanism. Instead of assigning a single token to an expert, Soft MoE performs a "soft assignment" through two primary steps:

  1. Dispatching: The model computes a set of "slots." Each slot is a weighted average (a convex combination) of all input tokens in a sequence. The weights are determined by a learnable parameter matrix and a softmax operation.
  2. Combining: After each expert processes its assigned slots, the final output tokens are reconstructed as a weighted average of the expert outputs.
Comparison of routing mechanisms in Sparse MoE (discrete assignment) versus Soft MoE (weighted average slots).

Because every operation is a matrix multiplication or a softmax, the entire routing mechanism is fully differentiable. This eliminates the need for sorting or top-k logic and removes the risk of "dropping" tokens. Every input token contributes fractionally to every slot, and every expert output contributes fractionally to every final token representation.

Experimental Evaluation

The researchers evaluated Soft MoE across several large-scale datasets, including JFT-4B, ImageNet-1k, and WebLI. The results demonstrate a significantly improved Pareto frontier regarding the trade-off between accuracy and computational efficiency compared to both standard Vision Transformers (ViTs) and existing sparse MoEs.

Soft MoE vs. Dense models on ImageNet finetuning accuracy.

Key Results

  • Efficiency: A Soft MoE B/16 model matched the performance of a massive ViT H/14 but was 5.7x faster at inference.
  • Scaling: The authors scaled Soft MoE up to a "Huge" configuration with 128 experts. This model had 40x more parameters than a standard ViT-H but only a 2% increase in inference time while providing substantially higher quality.
  • Training Speed: Soft MoE L/16 outperformed ViT H/14 while requiring only half the training time.

Discussion and Limitations

Soft MoE's hardware-friendly design avoids the communication bottlenecks typical of sparse MoEs by utilizing dense matrix operations. However, the primary limitation is its application to auto-regressive decoding (e.g., in LLMs). Because Soft MoE mixes all tokens in a sequence to create slots, it naturally breaks the causality required for language generation. While causal masking could be applied, efficient implementation of "Causal Soft MoE" remains a subject for future research.

Additionally, while inference time is low, the memory footprint remains high because all expert parameters must be stored in VRAM. This makes Soft MoE particularly suitable for high-throughput data center applications rather than edge devices with limited memory.

Summary updated

BibTeX
@inproceedings{puigcerver2024from,
  author = {Joan Puigcerver and Carlos Riquelme and Basil Mustafa and Neil Houlsby},
  title = {{From Sparse to Soft Mixtures of Experts}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=jxpsAj7ltE},
}
Figure