Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models

Feng Cheng, Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Li, Wei Xia

Introduction

Training deep learning models for video—such as those for action recognition or temporal detection—is a notorious memory hog. To capture long-term context, these models often need to process hundreds of frames simultaneously. The primary bottleneck isn't just the model parameters; it is the activations.

To perform standard backpropagation, a GPU must cache the intermediate activations of every layer for every frame during the forward pass so they can be used to calculate gradients during the backward pass. For a standard ResNet-50 processing 128 frames, this can easily exceed 40GB of VRAM, far beyond the capacity of most modern GPUs. Researchers often resort to "freezing" parts of the model or using pre-extracted features, but this prevents the model from learning optimal spatial features for the specific task at hand.

Stochastic Backpropagation (SBP)

The authors introduce Stochastic Backpropagation (SBP) to address this challenge. The core intuition is that while every frame is needed in the forward pass to build a complete temporal understanding, the backward pass (gradient calculation) doesn't need every single frame to update the model weights effectively. This is due to the high redundancy between adjacent video frames.

Comparison of training strategies: End-to-End, Freeze Backbone, Frame Selection, and Stochastic Backpropagation.

How it works:

  1. Full Forward Pass: The model processes all input frames to extract semantic information.
  2. Stochastic Backward Pass: For each layer, SBP randomly and independently removes a proportion of the backward paths based on a "keep-ratio" (e.g., 0.25).
  3. Memory Saving: Because the backward path for a specific node is dropped, the GPU no longer needs to cache the activations for that node during the forward pass.
  4. Efficient Implementation: To avoid complex memory management, the authors use a "re-forwarding" trick. During the initial forward pass, they only save the raw inputs for the sampled nodes. During the backward pass, they perform a tiny, local forward pass on just those nodes to re-generate the necessary activations for gradient calculation.

Key Insights: Exploiting Redundancy

A critical observation driving SBP is that activation maps for different frames are nearly identical in the bottom (early) layers of a network but diverge in the top (later) layers. SBP exploits this by primarily targeting the memory-heavy bottom layers where redundancy is highest.

Activation maps and their gradients across different layers, illustrating high redundancy in early layers.

Unlike "Frame Dropout," which removes frames from both the forward and backward passes, SBP keeps the forward pass intact. This allows the temporal modeling layers (the "brain" of the video model) to see the entire sequence, even if the spatial backbone (the "eyes") only receives sparse feedback. Interestingly, on smaller datasets like THUMOS, SBP with a low keep-ratio actually improved performance, suggesting that dropping gradients acts as a form of regularization, preventing the model from overfitting to redundant data.

Results and Performance

The authors evaluated SBP on major video benchmarks including Kinetics-400, EPIC-KITCHENS-55, and THUMOS’14. The results demonstrate that SBP is a highly effective strategy for memory-constrained environments:

  • Massive Memory Savings: On Video Swin-T, SBP reduced memory consumption from 15.2 GB to 4.4 GB (a ~70% saving) with a keep-ratio of 0.25.
  • Minimal Accuracy Trade-off: On Kinetics-400, the Top-1 accuracy drop was a negligible 0.6% (78.8% vs 78.2%).
  • Training Speedup: Despite the "re-forwarding" overhead, SBP resulted in a ~10% training speedup because the computationally expensive backward pass is significantly thinned out.
  • Democratizing Research: SBP allowed the researchers to train models end-to-end on 8x 16GB GPUs that previously required 8x 32GB GPUs, making high-end video research accessible to more modest hardware setups.

Limitations and Future Work

While the accuracy drop is small, it remains a factor. The authors note that SBP is a prototype for memory saving, and more sophisticated sampling strategies—such as adaptive layer-wise keep-ratios that drop more gradients in redundant early layers—could potentially close the gap. Future exploration into how SBP interacts with other techniques like Gradient Checkpointing and Mixed Precision at scale also remains an open area of study.

Summary updated

Abstract

We propose a memory efficient method, named Stochastic Backpropagation (SBP), for training deep neural networks on videos. It is based on the finding that gradients from incomplete execution for backpropagation can still effectively train the models with minimal accuracy loss, which attributes to the high redundancy of video. SBP keeps all forward paths but randomly and independently removes the backward paths for each network layer in each training step. It reduces the GPU memory cost by eliminating the need to cache activation values corresponding to the dropped backward paths, whose amount can be controlled by an adjustable keep-ratio. Experiments show that SBP can be applied to a wide range of models for video tasks, leading to up to 80.0% GPU memory saving and 10% training speedup with less than 1% accuracy drop on action recognition and temporal action detection.
BibTeX
@misc{cheng2022stochastic,
  author = {Feng Cheng and Mingze Xu and Yuanjun Xiong and Hao Chen and Xinyu Li and Wei Li and Wei Xia},
  title = {{Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models}},
  year = {2022},
  eprint = {2203.16755},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.16755},
}
Figure