Learning with Auxiliary Activation for Memory-Efficient Training

Sunghyeon Woo, Dongsuk JeonICLR 2023 · OpenReview

TL;DR

Training massive deep learning models is often limited by GPU memory, primarily because standard backpropagation requires storing every layer's input activations. This paper introduces Auxiliary Activation Learning (AAL), a novel training rule that uses "proxy" activations for weight updates. AAL allows for up to 15x memory compression with virtually no loss in accuracy and—crucially—without the training slowdowns typical of other memory-saving techniques.

The "Memory Wall" in Deep Learning

The "memory wall" is a well-known bottleneck in deep learning. When training a neural network using standard backpropagation (BP), the hardware must store the "input activations" for every single layer during the forward pass so they can be used later to calculate weight updates during the backward pass.

As models get deeper and resolutions get higher, this storage requirement explodes. Existing solutions usually come with heavy trade-offs:

  • Gradient Checkpointing: Saves memory by deleting activations and recomputing them later, but this significantly slows down training.
  • Reversible Networks: Architectures designed so activations can be calculated backward from the output, but these are restrictive and complex to design.
  • Activation Compression: Quantizes activations to lower precision, but often leads to accuracy degradation or slow "de-quantization" steps.

The authors sought a way to reduce memory usage significantly without sacrificing training speed or model performance.

Methods: Auxiliary Activation Learning (AAL)

The core idea of AAL is to decouple the activations used for the forward pass from the activations used for weight updates.

Comparison of conventional memory-saving algorithms and AAL

Instead of storing the exact input activation, the authors propose a new learning rule where the input to a layer is the sum of an "output activation" ($o_l$) and an "auxiliary activation" ($a_l$). While both are used to compute the next layer, only the auxiliary activation ($a_l$) is stored for the backward pass.

To ensure this actually works, the authors mathematically derived a Learning Criterion. They proved that as long as the proxy activation maintains a positive correlation with the real activation (specifically, a positive "Learning Indicator"), the model is guaranteed to converge.

Based on this, they propose two main flavors of this auxiliary activation:

The two variants of AAL: Auxiliary Residual Activation and Auxiliary Sign Activation
  1. Auxiliary Residual Activation (ARA): This reuses the input from a previous layer by leveraging residual connections. Since that data is already being stored for the skip connection, using it as the "proxy" for the current layer's update adds zero additional memory overhead.
  2. Auxiliary Sign Activation (ASA): Instead of storing a 32-bit floating-point activation, the model only stores its 1-bit sign (positive or negative). This reduces the memory footprint of that activation by 32x.

Key Insights

  • The "Good Enough" Update: The most profound insight is that backpropagation is surprisingly robust. You don't need the exact input activation to move the weights in the right direction; a "proxy" that captures the essential geometry of the activation space is sufficient for convergence.
  • Zero-Overhead Scaling: Unlike gradient checkpointing, which requires a second forward pass for every backward pass, AAL requires almost no extra computation. This means you get memory savings "for free" in terms of wall-clock time.
  • Orthogonality: AAL isn't a replacement for other methods; it’s a teammate. The authors showed that AAL can be combined with existing compression tools (like ActNN) or checkpointing to push memory savings even further than either could alone.

Experimental Results

The researchers put AAL through its paces across a wide variety of tasks and architectures, including ResNet, ViT, and MLP-Mixer for vision (CIFAR, ImageNet), as well as Transformer and BERT-Large models for NLP (IWSLT, MRPC, MNLI).

  • Performance Parity: In almost all tests, AAL (specifically the ARA variant) achieved accuracy within 0.1–0.2% of standard backpropagation. In some cases, like ResNet-152 on ImageNet, it actually slightly outperformed the baseline.
  • Massive Compression: When combined with ActNN, AAL achieved a 15.01x compression rate on ResNet-50.
  • Bigger Models on Same Hardware: Using AAL allowed the researchers to train models that were 13% deeper or 22% wider on a single 24GB GPU compared to standard backpropagation.
  • Speed: While methods like Mesa or ActNN often slow down training due to quantization overhead, AAL maintained nearly identical training times to the baseline.

Limitations and Future Work

While highly effective, the paper notes a few areas for improvement. The current implementation focuses on linear and convolutional layers and does not yet apply to normalization layers (like BatchNorm) or activation functions (like ReLU). Additionally, the "Sign" version (ASA) introduces a hyperparameter to scale the 1-bit activations, which might require tuning depending on the architecture. The authors plan to extend this logic to other types of operations to create a truly end-to-end memory-efficient framework.

Summary updated

Abstract

While deep learning has achieved great success in various fields, a large amount of memory is necessary to train deep neural networks, which hinders the development of massive state-of-the-art models. The reason is the conventional learning rule, backpropagation, should temporarily store input activations of all the layers in the network. To overcome this, recent studies suggested various memory-efficient implementations of backpropagation. However, those approaches incur computational overhead due to the recomputation of activations, slowing down neural network training. In this work, we propose a new learning rule which significantly reduces memory requirements while closely matching the performance of backpropagation. The algorithm combines auxiliary activation with output activation during forward propagation, while only auxiliary activation is used during backward propagation instead of actual input activation to reduce the amount of data to be temporarily stored. We mathematically show that our learning rule can reliably train the networks whose loss landscape is convex if the auxiliary activation satisfies certain conditions. Based on this observation, we suggest candidates of auxiliary activation that satisfy those conditions. Experimental results confirm that the proposed learning rule achieves competitive performance compared to backpropagation in various models such as ResNet, Transformer, BERT, ViT, and MLP-Mixer.
BibTeX
@inproceedings{woo2023learning,
  author = {Sunghyeon Woo and Dongsuk Jeon},
  title = {{Learning with Auxiliary Activation for Memory-Efficient Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=YgC62m4CY3r},
}
Figure