Sparse Mixture of Experts for Efficient Model Scaling

This direction is split into other directions: compute-capacity coupling. Its last version stays readable and citable here.

Version 1 of 1207 papers210 conceptsfrozen 2026-10-04

In standard Transformer architectures, there is a rigid, linear relationship between a model's knowledge capacity (total parameters) and its computational cost (FLOPs). To make a model "smarter," we typically add more parameters, but this forces every single parameter to be active for every input token. As we scale toward trillion-parameter models, this coupling becomes a "computational tax" that makes training and inference prohibitively expensive.

Sparse Mixture of Experts (MoE) breaks this coupling. It allows researchers to scale the total parameter count—the model's "internal library"—without a proportional increase in the work done per token.

The Core Mechanism: Conditional Computation

The intuition behind MoE is modularity. Instead of one giant feed-forward network (FFN), an MoE layer consists of many smaller, independent sub-networks called experts. For any given input token, a router (a lightweight gating network) decides which experts are best suited to process it.

As Nan Du and colleagues demonstrated with GLaM, this allows a model to have 1.2 trillion parameters while only activating a tiny fraction of them for each token. The result is a model that possesses massive internal knowledge but consumes significantly less energy and compute than a dense model of comparable performance. This "high-sparsity" approach is what enables models like Mixtral 8x7B to outperform much larger dense models like Llama 2 70B while using only a fraction of the active parameters during inference.

Solving the Routing Problem

The router is the most critical and fragile component of an MoE system. If the router is unstable, "representation collapse" can occur, where only a few experts are trained while others are ignored.

Researchers have developed several strategies to stabilize this:

  • Stable Routing: Damai Dai proposed StableMoE, which uses a two-stage process to first learn a fixed routing strategy and then train the experts, preventing the "moving target" problem where experts and routers try to learn simultaneously.
  • Soft Routing: To avoid the discrete, non-differentiable nature of picking "Top-K" experts, Joan Puigcerver and Neil Houlsby introduced Soft MoE. Instead of hard assignments, it uses a fully differentiable "soft" assignment that processes weighted averages of tokens, eliminating issues like token dropping.
  • Cartesian Routing: Xing Wu introduced CartesianMoE, which uses a multiplication-style routing mechanism to improve knowledge sharing between experts, making the model more robust to routing errors.

New Scaling Laws for Sparsity

The shift from dense to sparse architectures requires a new understanding of scaling laws. In dense models, performance is a function of parameters and data. In MoE, sparsity becomes a third dimension.

Aidan Clark and Diego de Las Casas proposed Unified Scaling Laws that introduce an "Effective Parameter Count," allowing direct comparisons between routed and dense models. Further work by Samira Abnar in Parameters vs FLOPs suggests that for a fixed training budget, increasing sparsity (more total experts, fewer active ones) consistently improves pretraining efficiency.

Beyond Feed-Forward Experts

While most MoE research focuses on replacing the FFN layers, the concept of "sparse lookup" is expanding to other parts of the architecture:

  • Memory Layers: Vincent-Pierre Berges and the Meta FAIR team scaled Memory Layers—trainable key-value lookup tables—to 128 billion parameters. This allows models to store factual knowledge in a massive external "memory" that is accessed sparsely, outperforming dense models with 4x the compute budget.
  • Latent N-Grams: Aurko Roy and colleagues introduced N-Grammer, which augments Transformers with a memory of discrete representations. By using sparse embedding lookups instead of dense matrix multiplications, they allow smaller models to match the quality of much larger ones.

Upcycling: From Dense to Sparse

A significant recent trend is "upcycling"—converting existing, expensive-to-train dense models into MoE models. Tingfeng Hui developed UpIT, which uses intermediate training checkpoints as specialized experts. Similarly, Yuchen Feng introduced DIVE, which discovers latent domain expertise by pruning models on different datasets, allowing for the reconstruction of a dense model into a diverse MoE with minimal retraining.

By treating a model as a collection of specialized modules rather than a monolithic block of parameters, sparse architectures provide a sustainable path toward trillion-parameter intelligence across domains—from language and vision to protein design and time series forecasting.

Papers in this version

Showing 60 of 207 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sparse_mixture_of_experts_for_efficient__10e1158a,
  title = {Sparse Mixture of Experts for Efficient Model Scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-mixture-of-experts-for-efficient-model-scaling/rdr_10e1158a0f334ab388622c3ba29dce36},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_sparse_mixture_of_experts_for_efficient__10e1158a_v1,
  title = {Sparse Mixture of Experts for Efficient Model Scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-mixture-of-experts-for-efficient-model-scaling/rdr_10e1158a0f334ab388622c3ba29dce36/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}