In conventional neural network architectures, a model's capacity—its ability to store knowledge and represent complex functions—is inextricably linked to its computational cost. When we scale up a standard dense model to capture more facts or handle diverse domains, we scale its parameter count. However, because every input token must interact with every single parameter during a forward pass, the computational requirement (measured in FLOPs) grows linearly alongside the parameter count.
This rigid coupling creates a severe bottleneck. As models approach the trillion-parameter scale, the financial, environmental, and hardware costs of executing full-parameter updates for every single token become unsustainable. The core insight of a rapidly growing body of research is that this coupling is unnecessary: we can build architectures that scale their capacity independently of their per-token computational footprint.
The Intuition of Conditional Computation
To understand why the dense scaling bottleneck exists, consider how a standard Transformer processes an input. Whether the model is processing a highly complex mathematical theorem or a simple punctuation mark, it routes the token through the exact same sequence of massive, dense matrix multiplications.
Decoupling capacity from compute relies on a principle known as conditional computation. Instead of activating the entire network for every input, the model dynamically activates only a small, relevant sub-network. The most prominent realization of this idea is the Mixture of Experts (MoE) architecture. Early foundational work by David Eigen et al. in Learning Factored Representations in a Deep Mixture of Experts demonstrated that stacking specialized sub-networks allows a model to represent an exponential number of specialized pathways while keeping active computational costs manageable.
Dense Architecture (Coupled) Sparse Architecture (Decoupled)
[ Input Token ] [ Input Token ]
│ │
▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼ ▼
┌───────────────────┐ ┌───────────────┐
│ All Parameters │ │ Router │
│ Are Active │ └───────┬───────┘
└───────────────────┘ │
│ ┌────────┴────────┐
▼ ▼ ▼
[ Output Token ] ┌─────────────┐ ┌─────────────┐
│ Expert A │ │ Expert B │
│ (Active) │ │ (Inactive) │
└──────┬──────┘ └─────────────┘
│
▼
[ Output Token ]
By replacing dense layers with a collection of "experts" and introducing a lightweight routing mechanism, the model can scale its total parameter count enormously while keeping the active parameters per token constant. For instance, Nan Du et al. demonstrated in GLaM: Efficient Scaling of Language Models with Mixture-of-Experts that a model can scale to 1.2 trillion parameters—7x the size of GPT-3—while only activating roughly 8% of its parameters per token. This architectural shift breaks the linear dependency between total capacity and inference latency.
Expanding Sparsity Beyond Feed-Forward Networks
While early sparse architectures focused primarily on partitioning the Feed-Forward Network (FFN) layers, recent innovations have extended the decoupling principle to other dimensions of the model architecture:
- Attention Heads: Standard multi-head attention requires every token to compute attention across all available heads. To decouple this, Zhang Xiaofeng et al. introduced the Mixture of Attention Heads (MoA) architecture, which dynamically selects a sparse subset of attention heads per token, scaling representational capacity without increasing the forward-pass computational footprint.
- Hidden Dimensions: Rather than treating the hidden-state dimensions as a uniform block, Yilong Chen et al. proposed the Mixture of Hidden-Dimensions (MoHD). This approach splits the hidden space into shared dimensions for common features and specialized dimensions that are dynamically routed, exploiting internal feature redundancies to save FLOPs.
- Memory and Residual Streams: The capacity of a Transformer's internal communication channel—the residual stream—is traditionally bound to the dimensions of its weight matrices. Brian Mak et al. bypassed this in Residual Matrix Transformers by replacing the vector-based stream with an outer product memory matrix, scaling memory capacity independently of compute. Similarly, Guillaume Lample et al. leveraged product quantization in Large Memory Layers with Product Keys to add a billion parameters via a sparsely-accessed lookup table with virtually zero computational overhead.
Algorithmic Challenges: Stability and Upcycling
Decoupling capacity from compute introduces unique algorithmic challenges, primarily centered around the routing mechanism. Because the router must learn which tokens to send to which experts, training can become highly unstable. If a few experts become overly popular, they receive all the gradient updates, leaving the remaining experts untrained—a failure mode known as representation collapse.
To enforce balance without relying on brittle, hard-to-tune auxiliary losses, Mike Lewis et al. developed BASE Layers: Simplifying Training of Large, Sparse Models. They formulated token routing as a classic linear assignment problem, mathematically guaranteeing that every expert processes an equal number of tokens. To combat the issue of tokens fluctuating between different experts throughout training, Damai Dai et al. introduced StableMoE: Stable Routing Strategy for Mixture of Experts, which learns a stable routing assignment in an initial phase and then freezes it, allowing experts to develop deep, consistent specialization.
Routing Optimization Strategies:
1. Linear Assignment (BASE Layers)
Tokens ──► [ Optimization Solver ] ──► Balanced Allocation (Equal load per expert)
2. Frozen Routing (StableMoE)
Tokens ──► [ Distilled & Frozen Router ] ──► Consistent Specialization (No routing drift)
Furthermore, training massive sparse models from scratch remains data-hungry and expensive. To circumvent this, researchers have designed methods to "upcycle" existing, well-trained dense models into sparse ones. Tong Zhu et al. demonstrated this with LLaMA-MoE, partitioning a pre-trained dense model's FFNs into sparse experts and applying targeted continual pre-training. To make this transition even more efficient, Tingfeng Hui et al. developed Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging, which utilizes intermediate dense training checkpoints as specialized experts, drastically reducing the data required to achieve high-performance sparse execution.
Generalization to Non-Text Modalities
The necessity of breaking the compute-capacity coupling is not unique to autoregressive language models. As deep learning architectures unify across modalities, the dense scaling bottleneck has reappeared in computer vision and generative modeling, prompting similar sparse solutions.
In computer vision, Carlos Riquelme developed V-MoE in Scaling Vision with Sparse Mixture of Experts, demonstrating that sparse routing applied to Vision Transformers allows models to scale to 15 billion parameters while cutting inference compute requirements in half.
This paradigm has also proven essential for scaling generative image models. Because high-fidelity image synthesis requires massive parameter counts to capture fine-grained textures and global layouts, dense Diffusion Transformers quickly become computationally prohibitive. Changqian Yu et al. addressed this in Scaling Diffusion Transformers to 16 Billion Parameters by introducing DiT-MoE. By routing image patches to sparse experts, the architecture successfully scales to 16.5 billion parameters while only activating 3.1 billion parameters per pass, proving that conditional computation is a universal mechanism for sustainable model scaling across all domains.
Key Foundational Work
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017) — contributed 1 concept elements to this direction.
Learning Factored Representations in a Deep Mixture of Experts (2014) — contributed 1 concept elements to this direction.