Doing more by doing less: how structured partial backpropagation improves deep learning clusters

Adarsh Kumar, Kausik Subramanian, Shivaram Venkataraman, Aditya Akella

Abstract

Training massive deep learning models is a resource-intensive process that often leaves expensive GPUs idle while waiting for network synchronization. This paper introduces Structured Partial Backpropagation (SPB), a technique where workers compute gradients for only a subset of model layers, and Jigsaw, a scheduler that packs these "partial" tasks to boost cluster efficiency. Evaluation shows that this approach improves cluster throughput by up to 28% with minimal impact on model accuracy.

The Efficiency Gap in Distributed Training

In large-scale distributed training, Data Parallelism is the standard: every GPU processes a batch of data, computes gradients for every layer, and synchronizes them over the network. While existing gradient compression techniques reduce network traffic, they do not reduce the compute or memory footprint on the GPU. To drop 80% of gradients, a system must still compute 100% of them first. Furthermore, traditional "gang scheduling" forces all workers to start and end at the same time, meaning any worker finishing a partial task early results in idle GPU time.

Methods: SPB and Jigsaw

The researchers propose a two-pronged solution to address these inefficiencies:

1. Structured Partial Backpropagation (SPB)

Instead of every worker computing gradients for all $L$ layers, each worker computes gradients for a "suffix" of the network (the layers closest to the output). Because the gradients of early layers depend on the results of later layers via the chain rule, compute savings are only possible by skipping the earliest layers.

Conceptual overview of Structured Partial Backpropagation (SPB) where different workers compute gradients for varying layer suffixes to save compute and memory.

This method exploits the fact that early layers in a DNN often converge faster than later, task-specific layers. Updating them less frequently has a negligible impact on final model quality while significantly reducing backward pass time and memory usage (as intermediate activations for skipped layers do not need to be stored).

2. Jigsaw Scheduler

Standard schedulers treat training jobs as monolithic blocks. Jigsaw, however, performs iteration-level scheduling. It recognizes that because of SPB, different workers have different task lengths. It "packs" these staggered tasks from multiple jobs onto the same GPUs, filling the gaps that would otherwise be idle time.

Jigsaw uses iteration-level DAG structures to pack staggered tasks from multiple jobs onto GPUs, maximizing cluster utilization.

Key Results and Insights

The authors evaluated model quality across Computer Vision (ResNet, VGG, DenseNet) and NLP (LSTM) domains, using a 45-GPU cluster simulation based on Microsoft training traces.

  • Cluster Efficiency: Jigsaw improved large-scale cluster efficiency (makespan and utilization) by 28% compared to state-of-the-art schedulers like Gandiva and Tiresias.
  • Resource Savings: For a ResNet-101 model, performing backprop on only the last 10% of layers reduced the backward pass time from ~80ms to ~8ms and cut peak memory usage in half.
  • Model Accuracy: Across 10 different models, the accuracy drop was minimal—usually less than 2%. For example, ResNet-18 on CIFAR-10 went from 94.02% (standard) to 93.96% (SPB).
  • Staggered Execution: In Jigsaw, the authors intentionally create "short" and "long" workers to create a scheduling puzzle that maximizes throughput, turning traditional "stragglers" into a feature for better packing.

Limitations and Future Work

While effective, SPB can show wider accuracy gaps on very complex tasks like CIFAR-100. The authors suggest "warm-up" epochs (initial training with full gradients) as a mitigation strategy. Additionally, the memory savings are most beneficial if the hardware stack can effectively reallocate freed memory to co-located tasks in real-time, which remains a systems challenge for future development.

Summary updated

BibTeX
@inproceedings{kumar2021doing,
  author = {Adarsh Kumar and Kausik Subramanian and Shivaram Venkataraman and Aditya Akella},
  title = {{Doing more by doing less: how structured partial backpropagation improves deep learning clusters}},
  booktitle = {DistributedML@CoNEXT},
  year = {2021},
  doi = {10.1145/3488659.3493778},
}
Figure