The Scaling Limits of Mixture of Experts

112 concepts93 papers1 proposal

Mixture-of-Experts (MoE) architectures offer an enticing proposition: scale a model's parameter count exponentially while keeping computational costs flat. By routing inputs to a sparse subset of specialized sub-networks ("experts"), MoE models theoretically decouple model capacity from compute.

However, empirical evidence across domains reveals a stark reality: you cannot scale the number of experts indefinitely. Whether in language modeling, computer vision, or speech processing, performance eventually plateaus or even degrades. As demonstrated in Towards a Comprehensive Scaling Law of Mixture-of-Experts, MoE performance is non-monotonic; expanding the expert pool or activation ratio beyond a critical threshold introduces structural distortion and harms accuracy.

To understand why the promise of infinite scaling breaks down, we must look at three interconnected bottlenecks: data starvation, routing complexity, and representational homogenization.

The Data Starvation Problem

The primary constraint on expert scaling is the data-to-parameter ratio. When adding more experts to a network, the incoming training data is partitioned into increasingly smaller, more specialized pools.

If the dataset size or task diversity remains fixed, each individual expert receives fewer tokens during training. As Bhosale et al. observe in Facebook AI's WMT21 News Translation Task Submission, low task diversity causes individual experts to suffer from insufficient token exposure. This prevents them from learning effective specialized representations, leading to slower convergence or outright performance declines.

Fixed Data Stream ───► [ Router ] ───┬───► Expert 1 (Ample data: generalizes well)
                                     └───► Expert 2 (Ample data: generalizes well)

Fixed Data Stream ───► [ Router ] ───┬───► Expert 1 (Sparse data: overfits)
                                     ├───► Expert 2 (Sparse data: overfits)
                                     ├───► Expert 3 (Sparse data: overfits)
                                     └───► Expert 4 (Sparse data: overfits)

Without massive data scaling, hyper-specialized experts quickly overfit to their tiny slivers of data. Tuzel et al. detailed this limitation in Robust Face Alignment Using a Mixture of Invariant Experts, showing that limited training data prevents the effective training of numerous specialized components. Similarly, Gross et al. found in Hard Mixtures of Experts for Large Scale Weakly Supervised Vision that while MoE architectures yield massive gains on web-scale datasets, they offer negligible benefits on smaller, curated datasets. When the data split becomes too granular, adding experts simply introduces noise, a phenomenon observed in recommendation adapters by Kong et al. in Customizing Language Models with Instance-wise LoRA for Sequential Recommendation and in person retrieval by the authors of DM-Adapter: Domain-Aware Mixture-of-Adapters for Text-Based Person Retrieval.

The Routing Bottleneck

Even when data is abundant, a larger expert pool complicates the task of the gating network, or router. The router must learn a highly complex, discrete classification function to map diverse inputs to the single best expert.

As the number of experts grows, this assignment task becomes increasingly fine-grained. Sam et al. explain in Divide and Grow: Capturing Huge Diversity in Crowd Images With Incrementally Growing CNN that while more experts raise the theoretical performance ceiling, the overall system plateaus because the classifier struggles to accurately assign inputs to a growing number of hyper-specialized sub-networks. If the router makes even minor errors, tokens are sent to sub-optimal experts, destroying the benefits of specialization. Furthermore, making the router itself larger or deeper does not solve the problem; Fu et al. found in MoE-SPNet: A Mixture-of-Experts Scene Parsing Network that increasing the complexity of gating functions yields diminishing returns.

This routing overhead interacts directly with the model's base capacity. Clark et al. established in Unified Scaling Laws for Routed Language Models that the benefits of adding experts diminish as the underlying dense representation grows larger, pointing to a theoretical parameter threshold where sparse activation gains are outweighed by routing complexity. If the candidate pool is too broad, the router's ranking ability degrades, causing performance drops at high activation levels, as noted by Gu et al. in Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts. To mitigate this, Zhang demonstrates in Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design that bounding routing flexibility by forcing experts to work in localized "squads" can actually improve accuracy by enforcing cleaner specialization boundaries.

Representational Homogenization

The final barrier to scaling is geometric: experts eventually run out of unique features to learn. Ideally, each expert should occupy a distinct niche in the model's representational space. However, as the number of experts increases, they begin to learn redundant information.

Cappellazzo et al. documented this saturation effect in Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach, showing that beyond a certain threshold, additional experts replicate existing knowledge rather than discovering new feature dimensions. This redundancy triggers what Li et al. describe in Theory on Mixture-of-Experts in Continual Learning as an extended exploration phase during training, which delays overall convergence without providing proportional gains in final performance.

Why can't we simply force experts to stay separate? Liu et al. explored this in Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer by designing an optimizer that forces expert parameter updates to be orthogonal to one another. They discovered a fundamental geometric limit: as the expert count grows, the aggregate parameter subspace becomes so dense and crowded that there is insufficient remaining orthogonal space for new experts to specialize uniquely. The experts are forced into homogenization.

Representational Space:

   [ Expert 1 ]          [ Expert 2 ]          [ Expert 1 ] [ Expert 3 ] [ Expert 2 ]

        [ Expert 3 ]                                   [ Expert 4 ] [ Expert 5 ]

   Low Expert Count: High Orthogonality        High Expert Count: Crowded Subspace
   (Clean, unique specializations)             (Homogenization and redundancy)

This ceiling manifests across a wide variety of specialized architectures. Whether partitioning the intermediate dimensions in fine-grained language models (Liao et al.), dividing spatial regression modes in computer vision (the authors of Mixture Dense Regression for Object Detection and Human Pose Estimation), or scaling cross-view transforms in person re-identification (the authors of Locally Aligned Feature Transforms across Views), performance consistently plateaus after a small number of experts.

Empirical Evidence Across Tasks

This saturation behavior is a shared property of the MoE paradigm across diverse applications:

The takeaway is clear: scaling Mixture-of-Experts models is not a matter of simply adding more parameters. True efficiency requires balancing the expert count against data volume, routing capacity, and the geometric limits of the representational subspace.

Key Foundational Work

Robust Face Alignment Using a Mixture of Invariant Experts (2016) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mixture_of_experts_scaling_limits_3597b85c,
  title = {mixture-of-experts scaling limits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mixture-of-experts-scaling-limits/rdr_3597b85c9cf8492592f2cd83c7274761},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_mixture_of_experts_scaling_limits_3597b85c_v1,
  title = {mixture-of-experts scaling limits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mixture-of-experts-scaling-limits/rdr_3597b85c9cf8492592f2cd83c7274761/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake