Backdrop: Stochastic Backpropagation

Siavash Golkar, Kyle CranmerarXiv 2018 · OpenReview

Abstract

We introduce backdrop, a flexible and simple-to-implement method, intuitively described as dropout acting only along the backpropagation pipeline. Backdrop is implemented via one or more masking layers which are inserted at specific points along the network. Each backdrop masking layer acts as the identity in the forward pass, but randomly masks parts of the backward gradient propagation. Intuitively, inserting a backdrop layer after any convolutional layer leads to stochastic gradients corresponding to features of that scale. Therefore, backdrop is well suited for problems in which the data have a multi-scale, hierarchical structure. Backdrop can also be applied to problems with non-decomposable loss functions where standard SGD methods are not well suited. We perform a number of experiments and demonstrate that backdrop leads to significant improvements in generalization.

Introduction

Stochastic Gradient Descent (SGD) and its variants are the engines of modern deep learning, but they rely on having a large number of independent samples to compute noisy, generalized gradients. This creates a bottleneck in two specific scenarios:

  1. Information-rich, low-sample data: In fields like cosmology or high-res medical imaging, datasets may consist of very few training samples, but each sample is massive and contains complex, multi-scale hierarchical information. Standard SGD cannot generate enough batch variance, and physically cropping the data destroys its macro-structure.
  2. Non-decomposable loss functions: Metrics like ROC AUC or F-measures require evaluating the entire dataset (or very large batches) at once. They cannot be broken down into a sum of per-sample losses, making small-batch SGD impossible or highly inaccurate.

Backdrop addresses these challenges by decoupling forward and backward stochasticity, allowing models to reap the generalization benefits of SGD even when processing large inputs or global losses.

The Backdrop Method

Backdrop is conceptually reminiscent of dropout but with a crucial twist: it only acts on the backward pass. It is implemented by inserting masking layers at specific points in the network.

Comparison of (a) Gradient Descent, (b) Minibatch SGD, and (c) Bernoulli Backdrop.

During the forward pass, the masking layer acts as a pure identity function. The network sees the entire, uncorrupted input, allowing it to accurately compute global, non-decomposable loss functions or process massive, uncropped images. During the backward pass, the layer randomly masks (drops) a percentage of the gradients flowing back through it, scaling the remaining gradients to preserve their expected norm. This mechanism injects the necessary noise into gradient updates to prevent overfitting without corrupting the forward signal.

Experimental Evaluation

The authors evaluated backdrop across several datasets, including CIFAR10 for non-decomposable losses, a multi-scale Gaussian Process (GP) texture generator, and the Ponce texture dataset for hierarchical learning.

Network architectures for Ponce, GP, and CIFAR datasets showing masking layer placement.

As shown in the architecture table, backdrop masking layers are typically placed after convolutional blocks. This allows the network to learn micro-features and macro-features simultaneously. For instance, in the GP texture experiments, applying backdrop at multiple scales improved classification accuracy from 45.9% to 66.2%, vastly outperforming physical image cropping which destroyed the multi-scale context.

Key Insights and Results

  • ROC AUC Optimization: On imbalanced CIFAR10 data, using backdrop with a massive batch size of 2048 outperformed standard SGD with small batch sizes.
  • Implicit Data Augmentation: For high-res images, backdrop applied to convolutional feature maps acts as stochastic patch-based training without destroying structural integrity.
  • Improved Confidence: While raw accuracy gains on standard CIFAR10 were modest, backdrop reduced test loss by up to 40%, leading to much more confident and well-calibrated model predictions.
  • Extreme Masking: Optimal performance often occurred at surprisingly high masking probabilities—sometimes dropping 94% to 98% of the gradients during a pass.

Conclusion

Backdrop provides a simple yet powerful mechanism for regularizing deep networks, particularly when standard minibatching is restricted by data size or loss complexity. While the high masking rates can increase training time, the technique offers a path toward optimizing complex, global objectives and learning from high-resolution, hierarchical data. Future work includes optimizing computational efficiency to skip masked gradient paths and exploring backdrop's impact on adversarial robustness.

Summary updated

Abstract

We introduce backdrop, a flexible and simple-to-implement method, intuitively described as dropout acting only along the backpropagation pipeline. Backdrop is implemented via one or more masking layers which are inserted at specific points along the network. Each backdrop masking layer acts as the identity in the forward pass, but randomly masks parts of the backward gradient propagation. Intuitively, inserting a backdrop layer after any convolutional layer leads to stochastic gradients corresponding to features of that scale. Therefore, backdrop is well suited for problems in which the data have a multi-scale, hierarchical structure. Backdrop can also be applied to problems with non-decomposable loss functions where standard SGD methods are not well suited. We perform a number of experiments and demonstrate that backdrop leads to significant improvements in generalization.
BibTeX
@misc{golkar2018backdrop,
  author = {Siavash Golkar and Kyle Cranmer},
  title = {{Backdrop: Stochastic Backpropagation}},
  year = {2018},
  eprint = {1806.01337},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.01337},
}
Figure