Reducing Activation Storage for Memory Efficient Training

52 concepts55 papers

In modern deep learning, the primary bottleneck for training large models is not the number of parameters, but the memory required to store intermediate activations. While a model's weights remain constant in size regardless of input, the "activation maps" generated during the forward pass scale linearly with both the depth of the network and the resolution or sequence length of the data.

Standard backpropagation (BP) requires these activations to be stored in memory until the error signal reaches them during the backward pass. This "locking dilemma" creates a significant hardware constraint, often preventing the training of deep architectures or high-resolution models on consumer-grade GPUs and edge devices. To overcome this, researchers have developed several strategies to reduce, compress, or entirely eliminate the need for activation storage.

Reversibility and Recalculation

One of the most direct ways to save memory is to avoid storing activations altogether by recomputing them on the fly. In standard networks, the transformation from one layer to the next is typically non-invertible, meaning information is lost and cannot be recovered without storage. However, by re-engineering the architecture to be mathematically reversible, the network can reconstruct the input of a layer from its output during the backward pass.

Sander et al. demonstrate this with Momentum Residual Neural Networks, which add a velocity term to standard residual blocks to ensure exact invertibility. This allows for training extremely deep models with a nearly constant ($O(1)$) memory footprint per layer. Similarly, Zhao et al. show that Multi-split Reversible Transformers can slash memory requirements by roughly 50% in neural machine translation, enabling deeper models on modest hardware. In the domain of high-resolution 3D imaging, Kellman utilizes reverse recalculation to reconstruct intermediate states, allowing massive physics-based networks to fit on consumer-grade hardware.

Localizing the Learning Signal

Another approach breaks the global backpropagation chain into smaller, independent modules. By training layers or blocks locally, the system only needs to store activations for a single module at a time rather than the entire network.

While local learning historically suffered from a "short-sightedness" that reduced accuracy, recent work has bridged this gap. Su et al. introduce HPFF, which uses overlapping cascade modules and patch feature fusion to achieve accuracy rivaling traditional BP while slashing memory usage by 80%. To further improve performance, Su et al. proposed the Momentum Auxiliary Network (MAN), which uses an exponential moving average to bridge the information gap between isolated blocks. Additionally, Ren et al. propose a BP-modified local loss that periodically uses global corrections to guide local training, achieving near-BP accuracy with 75% less memory.

Bypassing the Backward Pass

A more radical shift involves moving away from backpropagation entirely. Forward-only and zeroth-order (ZO) optimization methods estimate gradients or update weights using only forward passes, which naturally eliminates the activation storage bottleneck.

  • Forward-Forward and Equilibrium Prop: Gavves et al. optimize Equilibrium Propagation by training a fast initialization network to predict steady-state behavior, removing the need for global error signals. Reyes et al. and Park et al. explore the Forward-Forward Algorithm as a biologically plausible, memory-efficient alternative for hyperspectral imaging and federated learning.
  • Zeroth-Order Optimization: Methods like SubZero, proposed by Yu et al., estimate gradients within low-dimensional random subspaces. This allows for fine-tuning Large Language Models (LLMs) with a memory footprint nearly identical to inference. Chen et al. further enhance this with LOZO, which exploits the low-rank nature of LLM gradients to achieve near-first-order accuracy without the activation storage overhead.

Compressing the Intermediate State

If activations must be stored, they can be aggressively compressed. This is particularly useful for fine-tuning, where the backbone is frozen and only small adaptations are made.

Xu introduces DropIT, a technique that discards up to 90% of intermediate activation elements during training. By dropping the smallest-magnitude elements, the method acts as a noise-reduction mechanism that can actually improve final accuracy. For LLMs, the authors of VeLoRA project activations onto a fixed one-dimensional subspace, reducing storage to a tiny fraction of its original size.

Other methods focus on the precision of stored values. Yang et al. use 2-bit approximations for derivatives and share activation memory between adjacent layers, while Zhang et al. utilize Float8 training and kernel fusion to manage peak memory spikes in extreme classification tasks. By combining these architectural, algorithmic, and compression techniques, the research community is steadily dismantling the "memory wall" that has long constrained deep learning.

Key Foundational Work

Layer-Wise Invertibility for Extreme Memory Cost Reduction of CNN Training (2019) — contributed 1 concept elements to this direction.
Initialized Equilibrium Propagation for Backprop-Free Training (2019) — contributed 1 concept elements to this direction.
Non-discriminative data or weak model? On the relative importance of data and model resolution (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_backpropagation_memory_overhead_c049b217,
  title = {backpropagation memory overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backpropagation-memory-overhead/rdr_c049b2177b9a4628a1ab9465114f1cbd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_backpropagation_memory_overhead_c049b217_v1,
  title = {backpropagation memory overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backpropagation-memory-overhead/rdr_c049b2177b9a4628a1ab9465114f1cbd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake