Deploying deep learning models for inference on edge devices—like smartphones, smart home appliances, or microcontrollers—has become a standard practice. However, training or fine-tuning those same models on-site remains exceptionally difficult.
The primary barrier to edge learning is not the size of the model weights, nor is it strictly the computational complexity of the forward pass. Instead, the true culprit is the activation memory bottleneck. To understand why this bottleneck exists and how researchers are bypassing it, we must look at the fundamental asymmetry between inference and training.
The Asymmetry of Backpropagation
During inference, a neural network processes data sequentially. A layer takes an input tensor, computes an activation tensor, passes it to the next layer, and can then immediately purge that activation from random-access memory (RAM). The peak memory required for inference is roughly bounded by the size of the single largest layer's activation tensor plus the static model weights.
Training, however, is a two-way street. To calculate the gradient of the loss function with respect to a layer's weights during the backward pass, the chain rule requires access to the exact intermediate activations generated by that layer during the forward pass.
$$\frac{\partial \mathcal{L}}{\partial W_l} = \frac{\partial \mathcal{L}}{\partial z_l} \cdot (a_{l-1})^T$$
Because of this mathematical dependency, standard backpropagation forces the hardware to retain a history of all intermediate activations across the entire network until the backward pass reaches them. As highlighted by Alena Shilova et al. in Training on the Edge: The why and the how, this dynamic storage requirement frequently overwhelms the sub-megabyte SRAM capacities typical of low-power edge hardware.
Furthermore, architectures optimized specifically for low-parameter inference do not inherently solve this issue. Hung-Yueh Chiang et al. demonstrate in MobileTL: On-Device Transfer Learning with Inverted Residual Blocks that structural designs minimizing parameter counts often rely on expansion layers that generate massive intermediate feature maps. Alberto Ancilotto argues in XiNet: Efficient Neural Networks for tinyML that these large intermediate structures create a severe RAM bottleneck that starves concurrent application logic, making standard training pipelines unfeasible on microcontrollers.
To make on-device adaptation possible, recent research focuses on three core strategies: sparsifying the backward pass, compressing activation tensors, and bypassing backpropagation entirely.
Strategy 1: Sparsifying the Backward Pass
If storing every activation map is impossible, an intuitive solution is to selectively choose which activations to keep or update.
One approach is asymmetric sparsification. In Back Razor: Memory-Efficient Transfer Learning by Self-Sparsified Backpropagation, Ziyu Jiang et al. propose keeping the forward pass completely dense to preserve the network's representational accuracy, while aggressively pruning up to 97% of the activations stored for the backward pass. By computing gradients using these highly sparse activation maps, the peak memory footprint drops dramatically with negligible impact on final convergence.
Standard Backpropagation:
[Forward Pass] Layer 1 (Save A1) -> Layer 2 (Save A2) -> Layer 3 (Save A3)
[Backward Pass] Layer 1 (Use A1) <- Layer 2 (Use A2) <- Layer 3 (Use A3)
Asymmetric Sparsification (Back Razor):
[Forward Pass] Layer 1 (Save A1) -> Layer 2 (Prune A2) -> Layer 3 (Save A3)
[Backward Pass] Layer 1 (Use A1) <- Layer 2 (Sparse) <- Layer 3 (Use A3)
Another variant of this concept is the sparse update strategy. Instead of updating the entire network, optimization is restricted to a tiny subset of parameters or layers. Ji Lin et al. demonstrate in On-Device Training Under 256KB Memory that by combining a specialized Tiny Training Engine with sparse layer updates, complex deep learning models can be fine-tuned within a strict 256KB SRAM budget. This is complemented by compile-time graph pruning frameworks, such as the one introduced by Ligeng Zhu et al. in PockEngine: Sparse and Efficient Fine-tuning in a Pocket, which isolates and minimizes the exact activation dependencies required during backpropagation on mobile processors.
Strategy 2: Low-Rank Activation Compression
Rather than discarding activations, we can compress them into a highly compact, low-rank representation before storing them in RAM.
Kaushik Roy et al. introduce this paradigm in LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning. The authors utilize a "one-shot" tensor decomposition to project high-dimensional intermediate layer outputs into tiny, reusable subspaces. Instead of caching raw, memory-heavy feature maps, the device stores only these low-rank projections, expanding them back on-the-fly during the backward pass.
This spatial compression is frequently paired with numerical quantization. Storing high-precision floating-point activations creates a significant "memory wall" due to the energy and bandwidth required for off-chip data movement, as detailed by Seock-Hwan Noh et al. in LightNorm: Area and Energy-Efficient Batch Normalization Hardware for On-Device DNN Training. To mitigate this, Sifan Zhou et al. develop GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning, which replaces floating-point arithmetic entirely with a group-shared integer format. Quantizing both the forward activations and the backward gradients slashes memory consumption by half while keeping the numerical stability required for fine-tuning large language models.
Strategy 3: Bypassing Backpropagation Entirely
The most radical way to eliminate the activation memory bottleneck is to abandon backpropagation altogether. If gradients can be computed or estimated using only forward evaluations, the requirement to store historical activation maps disappears completely.
In Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach, Yequan Zhao et al. implement a quantized zeroth-order optimization method. Zeroth-order optimization estimates the gradient of a loss function by perturbing the weights slightly and observing the change in the output scalar:
$$g \approx \frac{\mathcal{L}(W + \mu u) - \mathcal{L}(W - \mu u)}{2\mu} u$$
Because this calculation relies solely on forward passes, the network can discard intermediate feature maps layer-by-layer exactly as it does during standard inference. This allows microcontrollers to fine-tune models using as little as 256 KB of RAM, transforming on-device learning into a process nearly as lightweight as inference.
This modern approach builds upon foundational concepts explored decades ago. Nikzad Benny Toomarian et al. originally laid the groundwork for forward-only gradient computation in Adjoint-Functions and Temporal Learning Algorithms in Neural Networks, leveraging adjoint operators to eliminate the massive memory overhead of backpropagation-through-time in dynamical systems.
Architectural Redesigns and Partitioning
When backpropagation cannot be avoided, researchers alter the network's execution flow or mathematical formulation to prevent peak memory spikes.
One method is submodule partitioning. As shown by Hong Jia et al. in TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices, breaking down a network into isolated submodules allows the device to run adaptation steps within strict 512 KB SRAM limits. The peak activation memory is constrained because only the activations of the active submodule need to be cached at any given moment.
Alternatively, modifying the interaction between layers can eliminate the generation of large activation maps. In DA3: Dynamic Additive Attention Adaption for Memory-Efficient On-Device Multi-Domain Learning, Li Yang et al. introduce a framework that shifts model updates from a multiplicative attention mechanism to an additive one. This mathematical shift ensures that the dimensions of the intermediate feature maps do not compound quadratically, removing the primary memory bottleneck during multi-domain edge adaptation.
By shifting the optimization focus from reducing static parameter counts to actively minimizing, compressing, or eliminating dynamic activation storage, these diverse methodologies collectively enable lifelong learning and real-time adaptation directly on resource-constrained hardware.
Key Foundational Work
Training on the Edge: The why and the how (2019) — contributed 1 concept elements to this direction.
Adjoint-Functions and Temporal Learning Algorithms in Neural Networks (1990) — contributed 1 concept elements to this direction.