As deep learning models scale to hundreds of billions or trillions of parameters, they inevitably collide with a physical reality: the memory capacity of a single accelerator is finite. When a model cannot fit on a single GPU, it must be distributed across a cluster of machines.
Historically, overcoming this "GPU memory wall"—a concept detailed by Rajbhandari et al. in ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning—required human experts to manually partition the model. Developers had to explicitly split tensors, insert low-level communication primitives, and orchestrate complex "3D parallelism" (combining data, tensor, and pipeline parallelism). As noted by Lai et al. in Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models, this manual approach is error-prone, labor-intensive, and tightly couples model code to specific hardware configurations.
To democratize large-scale training, a major research direction has emerged: automatic model partitioning. The core intuition is to shift the burden of distributed execution from the researcher to the system, transforming manual code refactoring into an automated optimization problem.
The Core Intuition: Neural Networks as Dataflow Graphs
Why can model partitioning be automated? The key lies in how deep learning frameworks represent neural networks: as directed acyclic graphs (DAGs) of computational operators, where nodes represent operations (like matrix multiplications) and edges represent the tensors flowing between them.
Instead of forcing a developer to manually annotate where to slice a model, automated systems analyze this computational graph. As Wang et al. demonstrated in Supporting Very Large Models using Automatic Dataflow Graph Partitioning, a system can use recursive search algorithms to automatically discover how to split operators across multiple devices. By analyzing tensor shapes and operator types, the framework can mathematically balance the computational load on each GPU while minimizing the communication overhead required to pass data between them.
This graph-centric view allows frameworks to be "non-intrusive." For example, Qararyah et al. developed a graph-partitioning method in A Computational-Graph Partitioning Method for Training Memory-Constrained DNNs that splits massive networks across devices without requiring any manual code rewrites from the user. The system takes a standard single-device model definition, slices the underlying graph, and injects the necessary synchronization points automatically.
Compiler-Driven Sharding
A powerful evolution of this idea is moving the partitioning logic directly into the machine learning compiler. Rather than treating distributed execution as a separate infrastructure layer, compiler-based frameworks treat sharding as a native optimization pass.
In GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, Lepikhin et al. introduced a lightweight sharding API that allows the underlying compiler to automatically determine how tensors should be distributed across thousands of accelerators. Similarly, the authors of AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism integrated sequence parallelism directly into the PyTorch 2.0 compiler stack. By doing so, the compiler automatically handles the complex inter-GPU communication protocols required for processing massive context lengths, matching the performance of highly specialized, hand-written libraries without the associated engineering overhead.
Co-Optimizing Space and Time
True automation cannot just decide where to split a model (spatial partitioning); it must also decide when to execute operations and transfer data (temporal scheduling). If a model is sliced poorly in time, GPUs will sit idle waiting for data from previous layers—a problem known as pipeline bubbles.
To solve this, modern frameworks co-optimize slicing and scheduling. In Pfeife: Automatic Pipeline Parallelism for PyTorch, Jhoo et al. developed a system that transparently slices models and schedules their execution using advanced prefetching techniques. This ensures that data arrives at the accelerator exactly when it is needed, outperforming manual frameworks.
Furthermore, automatic partitioning must be workload-aware. Spreading a small workload across too many GPUs can actually degrade performance due to communication bottlenecks. In Workload-aware Automatic Parallelization for Multi-GPU DNN Training, Shin demonstrated that by analyzing the computational graph beforehand, a system can predict the optimal number of devices to allocate, preventing power waste and communication overhead.
Extending the Memory Hierarchy
When a model grows so large that even a cluster of GPU memories is insufficient, automatic partitioning algorithms extend their search space beyond accelerators to include CPU RAM and non-volatile memory (NVMe).
Systems like KARMA, developed by Wahib et al., automatically orchestrate out-of-core training by combining data swapping with redundant recomputation. Similarly, Nie et al. introduced a fine-grained "Page" memory abstraction in Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent. Because these systems manage the memory hierarchy automatically, researchers can scale models to trillions of parameters while writing code as if they were training on a single machine.
Conclusion
The shift from manual model partitioning to automatic parallelization represents a critical maturation of machine learning infrastructure. By treating distributed training as a graph-partitioning and compiler-optimization problem, frameworks like Tofu, Merak, and Hetu (Miao et al. in Hetu: a highly efficient automatic parallel distributed deep learning system) allow developers to focus on algorithmic innovation rather than low-level hardware orchestration. The ultimate goal, as articulated in Minjie Wang's dissertation, Flexible and Efficient Systems for Training Emerging Deep Neural Networks, is a unified, transparent system where scaling up is as simple as running code locally.
Key Foundational Work
Piccolo: Building Fast, Distributed Programs with Partitioned Tables (2010) — contributed 1 concept elements to this direction.
Statistics Gathering for Learning from Distributed, Heterogeneous and Autonomous Data Sources (2003) — contributed 1 concept elements to this direction.