Overcoming the Activation Memory Bottleneck during Training

49 concepts46 papers5 proposals

To understand why training deep neural networks requires so much memory, we have to look closely at the mechanics of the backpropagation algorithm.

Consider a single linear layer performing a forward pass: it takes an input tensor $x$ (the activation from the previous layer) and multiplies it by a weight matrix $W$ to produce an output $y = Wx$. During the backward pass, the optimization algorithm needs to compute the gradient of the loss $L$ with respect to the weights ($\nabla_W L$) to update the model. By the chain rule, this gradient is calculated as:

$$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial y} \cdot x^T$$

Notice that the input activation $x$ appears directly in this equation. Because the backward pass runs in reverse order—from the final layer back to the first—the deep learning framework must cache every single intermediate activation tensor computed during the forward pass, holding it in memory until the backward pass finally visits that layer.

As models grow deeper, widths expand, and inputs scale (such as longer text sequences or higher-resolution images), this accumulated cache of intermediate tensors grows rapidly. As noted by Antoine Simoulin et al. in Memory-Efficient Fine-Tuning of Transformers via Token Selection and Zirui Liu et al. in Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model, intermediate activations frequently account for 70% to over 90% of total GPU memory usage during training, severely dwarfing the memory required for model parameters or optimizer states. This phenomenon is known as the activation memory bottleneck or the memory wall.

To bypass this wall, researchers have developed several distinct strategies that trade computation, precision, or architectural constraints for a drastically reduced memory footprint.

Trading Compute for Memory: Recomputation

The most direct way to eliminate activation storage is to throw the activations away immediately after computing them during the forward pass, and then recompute them on demand during the backward pass.

If a framework discards intermediate tensors, the GPU memory remains free. When the backward pass reaches a specific layer, the system performs a mini-forward pass from the nearest saved checkpoint to regenerate the missing activation just in time for the gradient calculation.

While this introduces extra computational overhead, the trade-off can be optimized systematically. For example, Xunyi Zhao et al. introduced Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch, which uses Integer Linear Programming (ILP) and Dynamic Programming to automatically determine the most efficient checkpointing schedules. Similarly, Jiawei Peng et al. leverage recomputation in Decoupled neural network training with re-computation and weight prediction to prevent memory explosions when training asynchronous, decoupled modules across distributed hardware.

Reversible Architectures: On-the-Fly Reconstruction

What if we could reconstruct past activations perfectly without running redundant forward passes from distant checkpoints? This is achieved by designing reversible networks, where each layer's input can be mathematically recovered from its output ($x = f^{-1}(y)$).

In a reversible architecture, the backward pass acts as a dual process: it simultaneously propagates gradients backward and reconstructs the layer inputs step-by-step from the final network output.

Early frameworks like MemCNN: a Framework for Developing Memory Efficient Deep Invertible Networks, developed by Sil C. van de Leemput et al., proved that this approach enables constant memory complexity relative to network depth. This paradigm has been successfully adapted to complex generative tasks by Tycho F. A. van der Ouderaa et al. in Reversible GANs for Memory-Efficient Image-To-Image Translation, and to structural feature extraction by Yuxuan Cai et al. in Reversible Column Networks.

To avoid the need to completely redesign standard architectures, JP Lewis demonstrated in On Exact Bit-level Reversible Transformers Without Changing Architecture that standard Transformer blocks can be made exactly bit-level reversible by treating them as ordinary differential equation (ODE) solvers, eliminating intermediate storage without altering inference behavior.

Lossy Storage: Activation Compression and Quantization

Because neural network training is inherently stochastic and resilient to minor perturbations, caching activations at full 32-bit or 16-bit floating-point precision is often unnecessary. Instead, activations can be compressed into lossy, low-bit representations before being stored in memory, and then decompressed during the backward pass.

Researchers have approached this via several distinct mechanisms:

Sparsification and Token Selection

In sequence-based architectures like Transformers, activation memory scales quadratically with sequence length due to the attention mechanism. However, not all tokens or activation values contribute equally to the final gradient updates. This insight has led to methods that selectively drop or approximate activation data.

For example, Changhyeon Lee et al. developed Softmax Output Approximation for Activation Memory-Efficient Training of Attention-based Networks. Instead of caching the massive, dense attention softmax matrix, their method stores only the highest and lowest values and approximates the remaining distribution during the backward pass using an exponential model.

Alternatively, one can filter data at the token level. Runjia Zeng et al. introduced TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching, which uses attention and gradient signals to identify and backpropagate through only the most salient tokens per instance, completely skipping activation caching for the omitted tokens.

Hybrid Differentiation and Targeted Freezing

A final class of techniques targets the activation bottleneck by altering how differentiation is applied or by freezing parts of the network entirely during parameter adaptation.

Nesting Forward-AD

While standard backpropagation relies entirely on reverse-mode automatic differentiation, Cong Guo proposed Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training. This approach embeds forward-mode automatic differentiation inside the standard backward pass specifically to handle complex activation functions (like GELU or Swish). Because forward-mode AD computes derivatives alongside the forward pass, it eliminates the need to cache the massive intermediate tensors typically required to differentiate these non-linear functions in reverse mode.

Parameter Freezing and Sidecars

When adapting large foundation models to downstream tasks, full backpropagation can be avoided by keeping the base model parameters static. Sid Wang et al. developed READ: Recurrent Adaptation of Large Transformers, which routes inputs through a lightweight, trainable RNN "sidecar" network while keeping the massive base Transformer frozen, avoiding the need to cache activations for the base model.

For scenarios where layers are selectively tuned, Arash Ardakani et al. introduced SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics. This framework tracks training dynamics to identify layers that have already converged, dynamically freezing them mid-training to systematically reduce the activation memory footprint as optimization progresses.

Key Foundational Work

MemCNN: a Framework for Developing Memory Efficient Deep Invertible Networks (2018) — contributed 1 concept elements to this direction.
Value-aware Quantization for Training and Inference of Neural Networks (2018) — contributed 1 concept elements to this direction.
Efficient Rematerialization for Deep Networks (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_activation_memory_bottleneck_training_me_6a2d9865,
  title = {activation memory bottleneck (training memory constraints)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-memory-bottleneck-training-memory-constraints/rdr_6a2d98650f4c46f0943c4457ecb16fa8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_activation_memory_bottleneck_training_me_6a2d9865_v1,
  title = {activation memory bottleneck (training memory constraints)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-memory-bottleneck-training-memory-constraints/rdr_6a2d98650f4c46f0943c4457ecb16fa8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake