PatrickStar: Parallel Training of Pre-trained Models via a Chunk-based Memory Management

Jiarui Fang, Yang Yu, Shenggui Li, Yang You, Jie Zhou

Introduction

Training Large Language Models (LLMs) like GPT-3 has traditionally been an "arms race" requiring massive supercomputing clusters. While initial pre-training often happens on hundreds of GPUs, the broader democratization of AI requires that researchers be able to fine-tune and experiment with these models on more accessible hardware, such as a single node with 8 GPUs. PatrickStar is a distributed training system designed to enable this by using a novel "chunk-based" dynamic memory management strategy.

The Challenge of Large Model Training

The primary bottleneck in training massive models is GPU memory. Existing solutions, such as ZeRO-Offload, utilize "heterogeneous training" by offloading some data to the much larger CPU RAM. However, these systems typically rely on static partitioning, deciding ahead of time which tensors reside on the CPU and which stay on the GPU.

This approach is fragile because "non-model data," such as intermediate activations, fluctuates significantly during training. If activations spike, the system often crashes with an Out-of-Memory (OOM) error, even if ample free space exists on the CPU. Furthermore, moving thousands of small tensors between devices is highly inefficient, failing to saturate the available interconnect bandwidth.

The PatrickStar Solution

PatrickStar introduces two core modules to address these inefficiencies: the Chunk-based Memory Manager (CMM) and the Dynamic Memory Manager (DMM).

Chunk-based Memory Manager (CMM)

Instead of managing thousands of individual tensors, PatrickStar organizes model data into large, fixed-size "chunks." This approach significantly improves bandwidth utilization because moving one large block of data is much faster than moving many tiny ones. It also simplifies memory fragmentation management, ensuring that the system can utilize every available byte of GPU memory.

Figure 3: PatrickStar organizes model data into chunks and manages their movement between CPU and GPU memory spaces.

Dynamic Memory Manager (DMM)

The DMM acts as the system's brain, treating CPU and GPU memory as a shared, fluid pool. It employs a Runtime Tracer during an initial "warm-up" iteration to record exactly how much memory activations consume at every step. Using this data, a Smart Eviction algorithm identifies which chunks won't be needed for the longest time and evicts them to the CPU to make room for incoming data. This allows the model data to "shrink" or "expand" its GPU footprint dynamically to accommodate the fluctuating needs of activations.

Key Results and Performance

PatrickStar's dynamic approach yields significant improvements in both model scale and training efficiency. By exploiting the repetitive nature of deep learning training, the system uses "future knowledge" from the warm-up phase to make optimal placement decisions.

Figure 2: Comparison of GPU memory usage over time, showing how PatrickStar's offloading strategy manages memory spikes compared to non-offloading methods.
  • Massive Model Scaling: On a single node, PatrickStar extended the maximum trainable model scale by 2.27x to 2.5x compared to DeepSpeed. For instance, on a V100 node where DeepSpeed could only handle a 4B parameter model, PatrickStar successfully trained a 12B parameter model.
  • Superior Throughput: PatrickStar consistently exhibited higher throughput (Tflops). On an 8-GPU cluster, it was roughly 1.37x faster than standard PyTorch DDP for smaller models and remained robust as model size increased.
  • GPT-3 Success: The system successfully ran a 175B parameter GPT-3 training task on a 32-GPU cluster, a feat that typically requires much larger infrastructure.

Conclusion

By moving away from static partitioning and embracing a chunk-based, dynamic memory management strategy, PatrickStar allows researchers to train models up to 2.5 times larger than previous state-of-the-art systems. While future work may involve integrating NVMe SSD offloading and further framework support, PatrickStar represents a significant step toward making large-scale AI training more accessible and efficient.

Summary updated

BibTeX
@article{fang2022patrickstar,
  author = {Jiarui Fang and Yang Yu and Shenggui Li and Yang You and Jie Zhou},
  title = {{PatrickStar: Parallel Training of Pre-trained Models via a Chunk-based Memory Management}},
  journal = {IEEE Transactions on Parallel and Distributed Systems},
  year = {2022},
  doi = {10.1109/tpds.2022.3219819},
}
Figure