Back Razor: Memory-Efficient Transfer Learning by Self-Sparsified Backpropagation

Ziyu Jiang, Xuxi Chen, Xueqin Huang, Xianzhi Du, Denny Zhou, Zhangyang WangNeurIPS 2022 · OpenReview

TL;DR

Back Razor is a "plug-and-play" framework that slashes the memory required to fine-tune deep learning models on edge devices. By using asymmetric sparsification—pruning up to 97% of the activations stored for the backward pass while keeping the forward pass dense—it achieves up to a 9.2x reduction in memory usage with virtually no loss in accuracy.

What Problem Does This Paper Address?

Deep learning is increasingly moving to the "edge" (phones, sensors, and IoT devices), but training models in these environments is a memory nightmare. While we have many techniques to make inference efficient (like weight pruning), training is much harder because of the activation bottleneck.

To calculate gradients during backpropagation, a framework must store the intermediate outputs (activations) of every layer from the forward pass.

Memory breakdown showing activations dominating the footprint

As illustrated above, for modern architectures like Vision Transformers (ViTs) or high-resolution CNNs, these stored activations can take up many times more memory than the model weights themselves. Existing solutions like gradient checkpointing save memory but significantly slow down training by re-computing layers, while "lite" modules often sacrifice accuracy.

Methods: Asymmetric Sparsification

To overcome the activation bottleneck, the authors introduce a novel approach called Asymmetric Sparsification.

Diagram of the Back Razor asymmetric sparsification process

The framework operates through four key steps:

  1. Dense Forward Pass: The model performs its forward computation exactly as usual. This ensures that the final prediction and the loss calculation remain highly precise.
  2. Sparse Storage: Instead of storing the full, dense activation tensors for the backward pass, Back Razor prunes them. It uses a simple Top-K magnitude pruning strategy: it keeps only the largest values (the most "important" signals) and sets the rest to zero.
  3. Sparse Backward Pass: During backpropagation, the gradients are calculated using these pruned, sparse activations.
  4. Efficient Format: To actually save memory, the sparse activations are stored as a bitmask (indicating non-zero positions) and a smaller dense matrix of the remaining values.

The core insight here is the realization that while the forward pass is highly sensitive to noise, the backward pass is remarkably robust. You can discard over 90% of the activation data used for gradient calculation, and the model will still converge to a high-quality solution.

Datasets Evaluated

The researchers evaluated Back Razor across a wide variety of tasks and architectures to ensure broad applicability:

  • Image Classification: Tested on eight popular transfer learning datasets including Pets, Aircraft, CIFAR-10/100, Flowers, Cars, CUB, and Food.
  • Semantic Segmentation: Pascal VOC using DeepLabV3-MobileNet.
  • Language Modeling: The RTE task from the GLUE benchmark using a BERT-base model.
  • Architectures: ProxylessNAS-Mobile (CNN) and ViT-B/16 (Vision Transformer).

Key Insights and Ideas

  • Asymmetry is the Secret Sauce: Previous attempts at activation pruning were "symmetric" (pruning both forward and backward). This led to error accumulation that tanked accuracy. By only pruning the data stored for the backward pass, Back Razor avoids this "drift."
  • Theoretical Convergence: The paper provides a mathematical proof showing that for CNNs, Back Razor maintains a convergence rate similar to vanilla Stochastic Gradient Descent (SGD) under mild conditions.
  • Breaking the ViT Barrier: Unlike some previous memory-saving methods (like TinyTL) that rely on freezing weights to avoid storing activations, Back Razor works effectively on Vision Transformers. This is crucial because ViTs' self-attention mechanisms require activations for gradient calculation even if weights are frozen.
  • Orthogonality: Back Razor can be combined with other techniques like Mesa (activation quantization) to push memory savings even further.

Key Results

  • Extreme Sparsity: On CNNs, the method can reach 97% to 99% sparsity in stored activations without degrading performance.
  • Massive Memory Savings:
    • For CNNs, it saved 9.2x memory compared to full fine-tuning.
  • For ViTs, it achieved a 3.9x theoretical saving, which increased to 7.5x when combined with quantization (Mesa).
  • No Accuracy Trade-off: In almost all tests, Back Razor matched or even slightly exceeded the accuracy of full-parameter fine-tuning (FT-Full) and significantly outperformed "lite" transfer methods like Bitfit.
  • Real-World Efficiency: On a GPU implementation, the authors measured a 2.7x actual on-device memory efficiency for ViTs.

Limitations and Future Work

  • Transformer Theory: While the empirical results for Vision Transformers are excellent, the current theoretical convergence proof doesn't fully cover the self-attention module because it isn't "Lipschitz continuous." The authors note this as a priority for future theoretical work.
  • Hardware Support: To translate 97% sparsity into 97% faster training or perfectly linear memory savings on all hardware, specialized sparse kernels are needed. The current implementation uses a bitmap/dense matrix workaround which is effective but could be further optimized at the compiler level.
  • Task Breadth: While they tested classification, segmentation, and one NLP task, further exploration into generative models (like GANs or Diffusion) or more complex NLP tasks remains open.

Summary updated

Abstract

Transfer learning from the model trained on large datasets to customized downstream tasks has been widely used as the pre-trained model can greatly boost the generalizability. However, the increasing sizes of pre-trained models also lead to a prohibitively large memory footprints for downstream transferring, making them unaffordable for personal devices. Previous work recognizes the bottleneck of the footprint to be the activation, and hence proposes various solutions such as injecting specific lite modules. In this work, we present a novel memory-efficient transfer framework called Back Razor, that can be plug-and-play applied to any pre-trained network without changing its architecture. The key idea of Back Razor is asymmetric sparsifying: pruning the activation stored for back-propagation, while keeping the forward activation dense. It is based on the observation that the stored activation, that dominates the memory footprint, is only needed for backpropagation. Such asymmetric pruning avoids affecting the precision of forward computation, thus making more aggressive pruning possible. Furthermore, we conduct the theoretical analysis for the convergence rate of Back Razor, showing that under mild conditions, our method retains the similar convergence rate as vanilla SGD. Extensive transfer learning experiments on both Convolutional Neural Networks and Vision Transformers with classification, dense prediction, and language modeling tasks show that Back Razor could yield up to 97% sparsity, saving 9.2x memory usage, without losing accuracy. The code is available at: https://github.com/VITA-Group/BackRazor_Neurips22.
BibTeX
@inproceedings{jiang2022back,
  author = {Ziyu Jiang and Xuxi Chen and Xueqin Huang and Xianzhi Du and Denny Zhou and Zhangyang Wang},
  title = {{Back Razor: Memory-Efficient Transfer Learning by Self-Sparsified Backpropagation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=mTXQIpXPDbh},
}
Figure