Abstract
Training massive deep learning models is a resource-intensive process that often leaves expensive GPUs idle while waiting for network synchronization. This paper introduces Structured Partial Backpropagation (SPB), a technique where workers compute gradients for only a subset of model layers, and Jigsaw, a scheduler that packs these "partial" tasks to boost cluster efficiency. Evaluation shows that this approach improves cluster throughput by up to 28% with minimal impact on model accuracy.
The Efficiency Gap in Distributed Training
In large-scale distributed training, Data Parallelism is the standard: every GPU processes a batch of data, computes gradients for every layer, and synchronizes them over the network. While existing gradient compression techniques reduce network traffic, they do not reduce the compute or memory footprint on the GPU. To drop 80% of gradients, a system must still compute 100% of them first. Furthermore, traditional "gang scheduling" forces all workers to start and end at the same time, meaning any worker finishing a partial task early results in idle GPU time.
Methods: SPB and Jigsaw
The researchers propose a two-pronged solution to address these inefficiencies:
1. Structured Partial Backpropagation (SPB)
Instead of every worker computing gradients for all $L$ layers, each worker computes gradients for a "suffix" of the network (the layers closest to the output). Because the gradients of early layers depend on the results of later layers via the chain rule, compute savings are only possible by skipping the earliest layers.
This method exploits the fact that early layers in a DNN often converge faster than later, task-specific layers. Updating them less frequently has a negligible impact on final model quality while significantly reducing backward pass time and memory usage (as intermediate activations for skipped layers do not need to be stored).
2. Jigsaw Scheduler
Standard schedulers treat training jobs as monolithic blocks. Jigsaw, however, performs iteration-level scheduling. It recognizes that because of SPB, different workers have different task lengths. It "packs" these staggered tasks from multiple jobs onto the same GPUs, filling the gaps that would otherwise be idle time.
Key Results and Insights
The authors evaluated model quality across Computer Vision (ResNet, VGG, DenseNet) and NLP (LSTM) domains, using a 45-GPU cluster simulation based on Microsoft training traces.
- Cluster Efficiency: Jigsaw improved large-scale cluster efficiency (makespan and utilization) by 28% compared to state-of-the-art schedulers like Gandiva and Tiresias.
- Resource Savings: For a ResNet-101 model, performing backprop on only the last 10% of layers reduced the backward pass time from ~80ms to ~8ms and cut peak memory usage in half.
- Model Accuracy: Across 10 different models, the accuracy drop was minimal—usually less than 2%. For example, ResNet-18 on CIFAR-10 went from 94.02% (standard) to 93.96% (SPB).
- Staggered Execution: In Jigsaw, the authors intentionally create "short" and "long" workers to create a scheduling puzzle that maximizes throughput, turning traditional "stragglers" into a feature for better packing.
Limitations and Future Work
While effective, SPB can show wider accuracy gaps on very complex tasks like CIFAR-100. The authors suggest "warm-up" epochs (initial training with full gradients) as a mitigation strategy. Additionally, the memory savings are most beneficial if the hardware stack can effectively reallocate freed memory to co-located tasks in real-time, which remains a systems challenge for future development.
Summary updated