Efficient Strong Scaling Through Burst Parallel Training

Seo Jin Park, Joshua Fried, Sunghyun Kim, Mohammad Alizadeh, Adam Belay

Abstract

Deep learning models are increasingly trained on massive GPU clusters, yet traditional scaling methods often hit an "efficiency wall." Weak scaling suffers from diminishing sample efficiency as batch sizes grow, while strong scaling leads to GPU underutilization due to small per-device workloads. We present DeepPool, a system that achieves efficient strong scaling through "burst parallelism"—dynamically varying GPU counts per layer—and "GPU multiplexing" to fill idle cycles with background tasks. DeepPool improves cluster throughput by up to 2.3× without compromising model accuracy.

Introduction: The Scaling Efficiency Wall

To accelerate training, researchers typically employ weak scaling, increasing the global batch size as more GPUs are added. However, larger batch sizes eventually reach a point of diminishing returns in sample efficiency, where the model learns less per image processed. Strong scaling, which keeps the global batch size fixed while increasing GPU count, avoids this but introduces a different problem: underutilization. Small per-GPU batches fail to saturate modern hardware, and communication overheads begin to dominate. DeepPool aims to make strong scaling efficient by reclaiming these wasted GPU cycles.

DeepPool: Exploiting Heterogeneous Scalability

The core insight behind DeepPool is that a neural network is not a monolithic computational block. Different layers exhibit vastly different scaling characteristics.

Figure 5: Speedup potential varies significantly across different layers of a neural network, justifying a dynamic allocation approach.

As shown in Figure 5, early layers often possess high internal parallelism and scale well, whereas later layers may see negligible speedup from additional GPUs. DeepPool leverages this through two main pillars:

  1. Burst Parallelism: A dynamic programming planner creates a "burst" schedule, allocating more GPUs to high-parallelism layers to finish them quickly and fewer GPUs to low-parallelism layers to maintain high efficiency.
  2. GPU Multiplexing: When the foreground job scales down for specific layers, DeepPool schedules low-priority background jobs (like hyperparameter searches) on the idle GPUs.

System Implementation and Optimization

DeepPool incorporates several low-level optimizations to ensure background tasks do not interfere with the critical training path. These include using CUDA Graphs to minimize CPU overhead, launch pacing to prevent high-priority kernels from being blocked, and real-time performance monitoring to pause background tasks if a slowdown is detected in the foreground job. This architecture ensures that the "bursty" nature of the primary training task is seamlessly integrated with secondary workloads.

Evaluation and Results

The system was evaluated using standard vision architectures (VGG-16, WideResNet-101-2, and Inception-v3) on large-scale clusters.

Figure 1: Comparison of speedup achieved by DeepPool's batch-optimal scaling versus traditional strong and weak scaling methods.

As illustrated in Figure 1, DeepPool's "batch-optimal scaling" significantly outperforms both traditional strong and weak scaling. Key results include:

  • Throughput Gains: A 1.2× to 2.3× improvement in total cluster throughput compared to standard data parallelism.
  • Foreground Acceleration: The main training job runs 11% to 38% faster than static allocation strategies for the same level of total cluster utilization.
  • Planner Efficiency: The burst parallel planner is highly efficient, finding optimal training plans for 1024 GPUs in just 0.05 to 3.23 seconds.

Conclusion and Future Work

DeepPool demonstrates that the inefficiencies inherent in strong scaling can be transformed into opportunities for higher cluster utilization. By treating model layers as heterogeneous units and multiplexing workloads, DeepPool provides a path toward faster, more efficient deep learning training on modern hardware. Future work will explore extending these techniques to model parallelism and multi-GPU background jobs, as well as investigating hardware-level preemption to further reduce interference between multiplexed tasks.

Summary updated

BibTeX
@inproceedings{park2022efficient,
  author = {Seo Jin Park and Joshua Fried and Sunghyun Kim and Mohammad Alizadeh and Adam Belay},
  title = {{Efficient Strong Scaling Through Burst Parallel Training}},
  booktitle = {MLSys},
  year = {2022},
}
Figure