When deep learning models began their rapid scaling trajectory, they quickly ran into a physical barrier: the memory capacity of a single hardware accelerator. In the early days of distributed training, the standard solution was Data Parallelism (DP). In a data-parallel setup, the entire model is replicated across every GPU in a cluster. Each GPU processes a different slice of the dataset, computes gradients locally, and synchronizes those gradients with its peers at the end of each training step.
Data parallelism works beautifully as long as the model fits comfortably inside a single device's memory. However, as models scaled from millions to billions of parameters, this approach hit what researchers call the "memory wall." As surveyed by Julia Gusak et al. in Survey on Efficient Training of Large Neural Networks, replicating the full model and optimizer state across every single node introduces massive memory redundancy. If a model requires 40 GB of memory just to hold its weights and training states, it cannot be trained on a 32 GB GPU—no matter how many thousands of GPUs you add to your cluster using standard data parallelism.
To train massive architectures, the paradigm had to shift from replicating the model state to sharding it.
Deconstructing Model State Memory
To understand why sharding works, we have to look at what actually occupies a GPU's memory during training. It isn't just the static model weights. For a standard transformer model trained with an advanced optimizer like Adam, the memory footprint is divided into several distinct components:
- Parameters: The static weights of the model.
- Gradients: The calculated updates for those weights.
- Optimizer States: The internal tracking vectors (like momentum and variance in Adam), which often consume twice as much memory as the parameters themselves.
- Activations: The intermediate layer outputs saved during the forward pass so they can be reused during the backward pass.
Instead of forcing every GPU to hold a complete copy of all four components, sharded model parallelism breaks these states apart and distributes them across the cluster.
This insight underpins modern state-sharding strategies like ZeRO (Zero Redundancy Optimizer) and Fully Sharded Data Parallelism (FSDP). As demonstrated in open-source large language model frameworks like OLMo by Dirk Groeneveld et al., sharding the optimizer states, gradients, and parameters across a GPU cluster allows the aggregate memory of the entire network to act as a single, unified pool. When a specific layer executes its forward or backward pass, the required shards are gathered on-demand via fast collective communication primitives and immediately discarded after use.
Intra-Layer and Multi-Dimensional Tensor Sharding
Slicing optimizer states across devices provides massive memory relief, but what happens when a single layer's weight matrix is so enormous that it cannot fit on a single GPU even during execution? This requires partitioning the model at the individual operator level, a technique known as Tensor Parallelism.
In a standard transformer layer, the primary computational bottleneck is matrix multiplication ($Y = XW$). Tensor parallelism splits the weight matrix $W$ across multiple devices. For instance, Xu Han et al. scaled CPM: A Large-scale Generative Chinese Pre-trained Language Model by partitioning the architecture along its width dimension.
Conceptually, this can be done in two ways:
- Column-wise splitting: The weight matrix is sliced vertically into columns ($W = [W_1, W_2]$). Each GPU multiplies the input $X$ by its respective column shard, producing independent slices of the output ($Y_1 = XW_1$ and $Y_2 = XW_2$). These slices are then concatenated.
- Row-wise splitting: The weight matrix is sliced horizontally into rows. The input $X$ must also be split to match. Each GPU computes a partial matrix multiplication, and an
All-Reduceoperation sums the results across devices to produce the final output.
To scale models even further, researchers have evolved from simple 1D linear partitioning to multi-dimensional grids. In GLaM, Nan Du et al. utilized a two-dimensional sharding algorithm to distribute weights and computation across an accelerator mesh.
By mapping GPUs to a multi-dimensional topology, memory usage scales more linearly with the number of devices. Qifan Xu developed an approach called Optimus, which partitions both activations and parameters across a 2D processor mesh using the SUMMA matrix multiplication algorithm, drastically reducing the memory redundancy found in 1D schemes. For extreme scales, frameworks like Tesseract arrange GPUs in a 3D grid, partitioning tensors across a new "depth" dimension to optimize throughput and minimize communication overhead.
Inter-Layer and Automated Graph Partitioning
An alternative axis for distributing a model is vertically, layer by layer, known as Pipeline Parallelism. Instead of splitting individual matrices, the network's layers are divided sequentially across a chain of devices. GPU 1 handles layers 1–10, GPU 2 handles layers 11–20, and so on. Naveen Arivazhagan et al. leveraged pipeline-parallel training in Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges to accommodate the diverse requirements of massive, multi-task translation architectures.
However, manually deciding where to cut a network or how to split complex tensors becomes incredibly difficult as model architectures grow more irregular. To solve this, systems researchers turned to automated dataflow graph partitioning.
In his dissertation, Flexible and Efficient Systems for Training Emerging Deep Neural Networks, Minjie Wang proposed partitioning computations at the fine-grained operator level rather than the coarse layer level. This approach allows for precise memory balancing across heterogeneous hardware.
This concept was realized in the Tofu system, where Minjie Wang et al. demonstrated in Supporting Very Large Models using Automatic Dataflow Graph Partitioning that an automated framework can recursively analyze a model's underlying dataflow graph. By automatically selecting the optimal partitioning strategy for each operator, it enables the training of models with memory requirements exceeding five times the capacity of a single accelerator, while maintaining near-ideal throughput.
The Communication Tax and Hybrid Topologies
If sharding model states is so effective, why not shard every model infinitely across thousands of devices? The limiting factor is the communication tax.
As Ilia Markov et al. note in Quantized Distributed Training of Large Models with Convergence Guarantees, while sharding solves local memory constraints, it introduces a significant synchronization penalty. Because parameters and gradients are scattered across the network, they must be frequently broadcast and reduced during every single forward and backward pass. If the network interconnect between nodes is slower than the computational speed of the GPUs, the hardware will sit idle, waiting for data transfers to complete.
+-----------------------------------------------------------------------+
| High-Speed Local Node (e.g., NVLink) |
| |
| +---------------+ Intra-Node Sharding +---------------+ |
| | GPU 1 |<===========================>| GPU 2 | |
| | Shard A State | | Shard B State | |
| +---------------+ +---------------+ |
+-----------------------------------------------------------------------+
||
|| Inter-Node Replica Sync
|| (Slower Network / InfiniBand)
||
+-----------------------------------------------------------------------+
| High-Speed Remote Node |
| |
| +---------------+ Intra-Node Sharding +---------------+ |
| | GPU 3 |<===========================>| GPU 4 | |
| | Shard A State | | Shard B State | |
| +---------------+ +---------------+ |
+-----------------------------------------------------------------------+
To balance memory relief with communication efficiency, modern large-scale training runs rely on tiered or hybrid topologies. For example, in Pretraining Billion-Scale Geospatial Foundational Models on Frontier, Aristeidis Tsaris et al. implemented a tiered sharding strategy.
Instead of sharding the model uniformly across the entire supercomputer, model states are fully sharded only within a local node, where ultra-high-bandwidth intra-node interconnects (like NVLink) can handle the rapid data gathering. Across different nodes, the system maintains standard model replicas synchronized via traditional data parallelism over slower inter-node connections. This hybrid approach successfully isolates the heavy communication overhead of state-sharding to high-speed local channels, providing the memory capacity needed to scale up to billions of parameters without sacrificing training efficiency.
Key Foundational Work
Fully distributed EM for very large datasets (2008) — contributed 2 concept elements to this direction.