Pipelined Backpropagation at Scale: Training Large Models without Batches

Atli Kosson, Vitaliy Chiley, Abhinav Venigalla, Joel Hestness, Urs Köster

Abstract

New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to explore the hardware and machine learning properties of alternative training algorithms. In this work we evaluate the use of small batch, fine-grained Pipelined Backpropagation, an asynchronous pipeline parallel training algorithm that has significant hardware advantages. We introduce two methods, Spike Compensation and Linear Weight Prediction, that effectively mitigate the downsides caused by the asynchronicity of Pipelined Backpropagation and outperform existing techniques in our setting. We show that appropriate normalization and small batch sizes can also aid training. With our methods, fine-grained Pipelined Backpropagation using a batch size of one can match the accuracy of SGD for multiple networks trained on CIFAR-10 and ImageNet. Simple scaling rules allow the use of existing hyperparameters for traditional training without additional tuning.

Introduction

Modern deep learning relies heavily on batch parallelism to hide memory latency on GPUs. However, this approach requires massive memory bandwidth and faces scaling limitations. An alternative is pipeline parallelism, where a model is sliced into sequential stages and distributed across different workers. When implemented at a "fine-grained" level (e.g., one layer per worker), workers can keep weights in local memory, drastically reducing bandwidth needs.

Standard pipeline parallelism typically requires "filling and draining" the pipeline for every batch to ensure mathematical consistency. This synchronous approach leaves workers sitting idle during the ramp-up and ramp-down phases, severely limiting hardware utilization. Pipelined Backpropagation (PB) addresses this by running asynchronously: workers update weights immediately without waiting for the pipeline to drain.

Comparison of synchronous pipeline parallelism (top) and asynchronous Pipelined Backpropagation (bottom). The asynchronous approach eliminates idle time (white boxes) but introduces gradient staleness.

While PB improves efficiency, it introduces two major mathematical challenges: stale gradients and weight inconsistency. By the time a gradient propagates back to a worker, the weights have already been updated multiple times. Furthermore, the forward and backward passes for a single input may use different versions of the model's weights.

Methods: Mitigating Asynchronicity

To make PB viable at a micro-batch size of 1, we introduce two novel compensation methods designed for use with Stochastic Gradient Descent with Momentum (SGDM).

Spike Compensation (SC)

Momentum typically smooths gradients over time. In an asynchronous pipeline, the delay in gradient arrival disrupts this smoothing. Spike Compensation modifies the weight update equation to immediately apply the "missing" contribution of a delayed gradient. By injecting a mathematical "spike" into the impulse response, the algorithm counteracts the delay and restores the intended momentum dynamics.

Visualization of Spike Compensation. The delayed impulse response (middle) is corrected by injecting a spike (right) to restore the effective momentum of the system.

Linear Weight Prediction (LWP)

To address weight inconsistency, where the forward pass uses outdated weights relative to the eventual backward pass, we utilize Linear Weight Prediction. LWP uses the optimizer's current velocity to predict future weight values. By using these predicted weights during the forward pass, the system minimizes the discrepancy between the model states used in the two halves of the training step.

Combined Mitigation and Scaling

Combining LWP and SC allows the system to tackle both inconsistency and delay simultaneously. Furthermore, because we utilize a batch size of 1, we apply mathematical scaling to the learning rate and momentum from standard large-batch baselines. This ensures the training trajectory remains stable and allows the use of existing hyperparameters without extensive re-tuning.

Key Insights

Our research highlights several counter-intuitive findings in asynchronous training:

  1. Small Batches as a Feature: Using a batch size of 1 actually aids asynchronous training. Because weights change minimally between individual samples, the negative impacts of gradient delay and weight inconsistency are naturally reduced.
  2. Gradient Delay is the Primary Hurdle: Contrary to prior assumptions that focused on weight inconsistency (often addressed via memory-heavy "Weight Stashing"), we found that stale gradients are the primary cause of accuracy loss in fine-grained PB.
  3. Normalization Stability: Standard Batch Normalization fails at a batch size of 1. We found that replacing it with Online Normalization (ON) provided significantly more stability for PB training than other alternatives like Group Normalization.

Results

The combined $LWP_D + SC_D$ method successfully recovered full SGDM accuracy across multiple benchmarks. On ImageNet using ResNet-50, our combined PB method achieved 75.8% accuracy, slightly exceeding the 75.7% synchronous baseline. On CIFAR-10, our methods outperformed existing asynchronous mitigation strategies such as PipeDream, PipeMare, and Gradient Shrinking. Notably, our approach achieves these results with minimal memory and compute overhead compared to prior state-of-the-art methods like SpecTrain.

Conclusion

By introducing Spike Compensation and Linear Weight Prediction, we demonstrate that fine-grained Pipelined Backpropagation can match the accuracy of standard synchronous training even with a batch size of one. While current GPUs are optimized for large-batch processing, these algorithmic advancements pave the way for highly efficient training on next-generation hardware architectures, such as Coarse-Grained Reconfigurable Arrays (CGRAs) and Wafer-Scale Engines, where local memory and fine-grained parallelism are paramount.

Summary updated

Abstract

New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to explore the hardware and machine learning properties of alternative training algorithms. In this work we evaluate the use of small batch, fine-grained Pipelined Backpropagation, an asynchronous pipeline parallel training algorithm that has significant hardware advantages. We introduce two methods, Spike Compensation and Linear Weight Prediction, that effectively mitigate the downsides caused by the asynchronicity of Pipelined Backpropagation and outperform existing techniques in our setting. We show that appropriate normalization and small batch sizes can also aid training. With our methods, fine-grained Pipelined Backpropagation using a batch size of one can match the accuracy of SGD for multiple networks trained on CIFAR-10 and ImageNet. Simple scaling rules allow the use of existing hyperparameters for traditional training without additional tuning.
BibTeX
@inproceedings{kosson2021pipelined,
  author = {Atli Kosson and Vitaliy Chiley and Abhinav Venigalla and Joel Hestness and Urs Köster},
  title = {{Pipelined Backpropagation at Scale: Training Large Models without Batches}},
  booktitle = {MLSys},
  year = {2021},
  url = {https://openreview.net/forum?id=VMdD3xJX5PH},
}
Figure