Sheared Backpropagation for Fine-Tuning Foundation Models

Zhiyuan Yu, Li Shen, Liang Ding, Xinmei Tian, Yixin Chen, Dacheng Tao

Sheared Backpropagation for Fine-Tuning Foundation Models

Abstract

Fine-tuning large-scale foundation models is often constrained by the high memory overhead of activations stored during the forward pass. While activation pruning offers a potential solution, traditional unstructured methods frequently introduce computational overhead that offsets memory savings. This paper introduces PreBackRazor and its enhanced version PreBackRazor++, a novel activation pruning scheme that utilizes hardware-friendly "sheared" (column-wise) sparsity. By employing a bandit-based adaptive selection mechanism, the proposed method achieves up to a 1.7x speedup and a 2.9x reduction in memory usage on consumer-grade GPUs while maintaining baseline accuracy across vision and language tasks.

Introduction

Adapting massive Transformer-based models to specific downstream tasks via fine-tuning is a standard practice in modern AI. However, the memory requirements for storing intermediate activations—necessary for gradient calculations during backpropagation—often exceed the capacity of consumer-grade hardware. Previous approaches like the BackRazor algorithm attempted to mitigate this through "top-k" element-wise pruning. Unfortunately, these methods suffer from a "memory-for-compute" trade-off: the time required to identify and sort small activation values often negates the speed gains from sparsity. Furthermore, unstructured sparsity is difficult to accelerate using standard GPU kernels.

To address these challenges, we present PreBackRazor, which shifts from element-wise to structured, column-wise pruning. This approach aligns with modern linear algebra kernels, enabling actual wall-clock speedups alongside significant memory reductions.

Methodology

The core of PreBackRazor lies in its ability to predict and prune non-impactful activations before they are fully computed or stored.

Overview of the PreBackRazor architecture, illustrating the integration of sheared sparsity and the JointSpar adaptive selection mechanism.

The method is built upon four key strategies:

  1. Sheared (Column-wise) Sparsity: Pruning entire columns of activations ensures compatibility with standard BLAS kernels, facilitating hardware acceleration.
  2. Bandit-based Adaptive Selection: Inspired by the JointSpar algorithm, a multi-armed bandit maintains a probability distribution over activation coordinates. This allows the model to "sample" and decide which coordinates to freeze prior to the forward pass.
  3. Asymmetric Backpropagation: The system maintains a dense forward pass to preserve feature quality but utilizes a sparsified, sheared version of activations for the backward pass.
  4. FlashAttention Integration (PreBackRazor++): The authors customized the FlashAttention-2 kernel to support structured pruning, allowing the attention mechanism to benefit from efficiency gains without additional memory overhead.

Experimental Evaluation

The researchers evaluated PreBackRazor and PreBackRazor++ across several benchmarks, including ViT-B/16 (pre-trained on ImageNet-22K) and BERT-base. Tasks included image classification (CIFAR-10, CIFAR-100, CUB-200) and natural language inference (RTE task from GLUE).

Key Results

The experimental results demonstrate that structured sparsity is significantly more effective than unstructured alternatives.

Performance comparison of PreBackRazor++ against full fine-tuning and the original BackRazor method across different GPU architectures.
  • Computational Speedup: On an NVIDIA RTX 4090, PreBackRazor++ achieved a 1.7x speedup compared to the original BackRazor method when integrated with FlashAttention.
  • Memory Efficiency: Memory allocation was reduced by approximately 2.9x compared to full fine-tuning (e.g., reducing ViT training memory from ~20GB to ~6.8GB).
  • Accuracy Preservation: Despite pruning up to 90% of activations, the models maintained accuracy levels comparable to full fine-tuning. Notably, on the BERT-RTE task, PreBackRazor achieved the highest accuracy (72.92%) among all tested methods.

Discussion and Future Work

PreBackRazor demonstrates that predictive, structured pruning can overcome the limitations of traditional activation compression. By aligning algorithmic sparsity with hardware architecture, it enables efficient fine-tuning on consumer-grade GPUs.

However, several areas for future exploration remain. The current theoretical convergence analysis is primarily focused on linear networks; extending this to the non-linear dynamics of Transformers is a priority. Additionally, while the method shows promise for models like ViT and BERT, its scalability to Large Language Models (LLMs) with tens of billions of parameters requires further validation. Finally, deeper custom CUDA kernel optimization could further enhance the performance of the sheared backpropagation pattern.

Summary updated

Abstract

Fine-tuning is the process of extending the training of pre-trained models on specific target tasks, thereby significantly enhancing their performance across various applications. However, fine-tuning often demands large memory consumption, posing a challenge for low-memory devices that some previous memory-efficient fine-tuning methods attempted to mitigate by pruning activations for gradient computation, albeit at the cost of significant computational overhead from the pruning processes during training. To address these challenges, we introduce PreBackRazor; a novel activation pruning scheme offering both computational and memory efficiency through a sparsified back-propagation strategy, which judiciously avoids unnecessary activation pruning and storage and gradient computation. Before activation pruning, our approach samples a probability of selecting a portion of parameters to freeze, utilizing a bandit method for updates to prioritize impactful gradients on convergence. During the feed-forward pass, each model layer adjusts adaptively based on parameter activation status, obviating the need for sparsification and storage of redundant activations for subsequent backpropagation. Benchmarking on fine-tuning foundation models, our approach maintains baseline accuracy across diverse tasks, yielding over 20% speedup and around 10% memory reduction. Moreover, integrating with an advanced CUDA kernel achieves up to 60% speedup without extra memory costs or accuracy loss, significantly enhancing the efficiency of fine-tuning foundation models on memory-constrained devices.
BibTeX
@inproceedings{yu2024sheared,
  author = {Zhiyuan Yu and Li Shen and Liang Ding and Xinmei Tian and Yixin Chen and Dacheng Tao},
  title = {{Sheared Backpropagation for Fine-Tuning Foundation Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00562},
}
Figure