Activation Compression and Mixed Precision for Memory Efficient Training

574 concepts358 papers8 proposals

When training a deep neural network, the hardware memory bottleneck is rarely the model weights themselves. Instead, it is the activation memory—the intermediate layer outputs generated during the forward pass that must be retained in memory until the backward pass computes their corresponding gradients.

Because activation memory scales linearly with network depth, batch size, and sequence length, it frequently triggers out-of-memory errors on modern hardware. To break through this "memory wall," a prominent research direction focuses on reducing the footprint of these stored tensors. This is achieved through two complementary strategies: mixed-precision arithmetic, which lowers the bit-width of active computations, and activation compression, which quantizes or projects intermediate states into low-dimensional spaces specifically for storage.

The Core Conflict: Compute Precision vs. Accumulation Precision

The intuition behind mixed-precision training stems from a simple observation: neural networks are remarkably robust to noise during forward propagation, but highly sensitive to noise during gradient accumulation.

Early work by Matthieu Courbariaux et al. in Low precision arithmetic for deep learning established that we can decouple the precision used for matrix multiplications from the precision used to update weights. This concept was formalized into modern training pipelines by Paulius Micikevicius et al. in Mixed Precision Training.

The standard framework operates via three core mechanisms:

  1. Half-Precision Execution: The forward and backward passes execute using 16-bit floating-point formats (FP16 or BF16), which slashes the memory footprint of activations and weights by half and leverages high-throughput tensor cores.
  2. Master Weights: Because small gradient contributions can round to zero in 16-bit formats, a master copy of the weights is maintained in 32-bit single-precision (FP32). Gradients computed in FP16 are accumulated directly into these FP32 master weights.
  3. Dynamic Loss Scaling: Small gradient values risk underflowing the representable range of FP16. To prevent this, the loss is multiplied by a scaling factor prior to backpropagation, shifting the gradient distribution into a valid numerical range. The factor is dynamically adjusted to maximize precision without causing numerical overflow.
[FP32 Master Weights] ──(Cast)──> [FP16 Weights] ──> [Forward Pass (FP16 Activations)]
         ▲                                                       │
         │                                                    (Loss)
  (Accumulate Update)                                            │
         │                                            (Apply Loss Scaling)
         │                                                       ▼
 [FP16 Gradients] <────────────────────────────────────── [Backward Pass]

Activation Quantization: Compressing the Forward Pass

While mixed-precision training halves memory usage, activation maps still dominate the hardware footprint in high-throughput workloads, as noted by Asit K. Mishra et al. in WRPN: Wide Reduced-Precision Networks. To compress these maps further, researchers developed techniques to store activations in ultra-low-bit integer formats during the forward pass, reconstructing them only when needed for the backward pass.

In Backprop with Approximate Activations for Memory-efficient Network Training, Ayan Chakrabarti et al. demonstrated that the forward pass can remain exact while the activations saved for the backward pass are compressed into 4-bit or 8-bit integers. This asymmetric approach ensures that the model's forward predictions are uncorrupted, while the backward pass tolerates the slight approximation noise introduced by dequantization.

To push compression boundaries even further without sacrificing accuracy, subsequent frameworks introduced two key insights:

Low-Rank Projections: Subspace Backpropagation

For resource-constrained environments like microcontrollers or edge devices, quantization alone may not suffice. A parallel paradigm bypasses full-resolution storage entirely by projecting high-dimensional activations into a low-rank subspace during the forward pass.

The mathematical intuition is that neural network activations are highly redundant and reside on a lower-dimensional manifold. Instead of saving the full activation tensor $A$, the system computes and stores a compressed core tensor or factor matrices.

Forward Pass:  High-Dim Activation (A) ──(Low-Rank Projection)──> Core Tensor (C) [Stored]
                                                                        │
Backward Pass: Gradient (∇A) <───(Subspace Reconstruction)──────────────┘

In Memory-Efficient Fine-Tuning via Low-Rank Activation Compression, Jiang-Xin Shi et al. utilized a sampling-based decomposition to break intermediate activations into low-rank factor matrices on the fly, reconstructing them during gradient computation. For extreme hardware limits, Kaushik Roy et al. developed LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning, which employs a fixed, one-shot tensor projection. By executing the backward pass directly on these compressed core tensors, it eliminates full-resolution feature maps from SRAM entirely.

This subspace optimization can also be applied to the parameter space. Yiming Chen et al. demonstrated in A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models that optimizing model weights within a randomized, low-rank projected subspace allows the system to store only the projected activations. Because the gradient calculation for the subspace parameters depends exclusively on these low-dimensional projections, the full-rank activation memory bottleneck is completely bypassed.

Hybrid Deployments and Edge Learning

The convergence of mixed precision and activation compression has expanded the boundaries of where and how deep learning models can be trained.

On one end of the spectrum, these techniques enable the training of massive, complex architectures. For example, the reproduction of AlphaFold3 by Xinshi Chen et al. in Protenix relies heavily on BF16 mixed-precision training to scale biomolecular structure prediction. On the other end, they allow low-power IoT devices to perform on-device adaptation. As explored by Magnus Almgren in MicroTL: Transfer Learning on Low-Power IoT Devices, a hybrid precision architecture can freeze a quantized, integer-based backbone for feature extraction while maintaining a floating-point classification head for stable gradient descent, bringing localized learning to the edge.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mixed_precision_training_dfe4dd9f,
  title = {mixed-precision training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mixed-precision-training/rdr_dfe4dd9fc746494ab68e948f1c5da7db},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_mixed_precision_training_dfe4dd9f_v1,
  title = {mixed-precision training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mixed-precision-training/rdr_dfe4dd9fc746494ab68e948f1c5da7db/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake