Deep learning models are typically trained using backpropagation (BP), a process that requires two distinct phases: a forward pass to compute predictions and a backward pass to update weights. To calculate the necessary gradients during the backward pass, the system must store every intermediate "activation" (the output of each layer) from the forward pass.
On high-end GPUs, this is manageable. But on edge devices—smartphones, IoT sensors, and microcontrollers—this requirement creates a "memory wall." As Mengwei Xu and colleagues note in FwdLLM, the memory footprint of these stored activations can be three to four times larger than the model itself, often exceeding the physical RAM of mobile hardware. To enable on-device personalization and real-time adaptation, researchers are developing a suite of "backpropagation-free" and memory-efficient training mechanisms.
Zeroth-Order Optimization: Learning via Perturbation
One of the most promising ways to bypass the memory wall is to stop calculating gradients altogether. Instead of using calculus to find the exact direction to update weights, Zeroth-Order (ZO) optimization estimates the gradient by slightly "poking" the model and seeing how the output changes.
In frameworks like MobiZO, developed by Amir Ziashahabi and others, the system performs two forward passes: one with the original weights and one with a small amount of random noise (perturbation) added to the weights. By comparing the loss between these two passes, the optimizer can infer which direction the weights should move. Because this only requires forward passes, the device never needs to store intermediate activations for a backward pass. Haozhe Feng demonstrates in BAFFLE that this approach makes federated learning viable even for "black-box" environments where internal gradients are inaccessible.
Forward-Only and Local Learning
While ZO optimization avoids the backward pass by treating the model as a black box, Forward-Only (FO) algorithms break the model into independent, trainable pieces. Standard BP is "globally locked"—no layer can update until the final error is calculated at the very end and sent all the way back.
Researchers like Amir Aminifar have proposed alternatives like Bio-BPfree and TinyFoA. These methods use local loss functions at each layer. Instead of waiting for a global signal, each layer tries to optimize its own representation—often by maximizing the "goodness" or distance between different classes of data—immediately after its forward pass. By "slicing" the network vertically or using module-wise updates, these algorithms allow activations to be discarded as soon as the next layer is computed, slashing memory overhead by up to 96x.
Synthetic Gradients and Asynchronous Pipelines
Another approach involves "predicting" the future. In Feed-Forward On-Edge Fine-Tuning Using Static Synthetic Gradient Modules, Robby Neven and his team introduce Synthetic Gradient Modules (SGMs). These are small, auxiliary networks that learn to predict what the gradient would have been without actually performing the backward pass. This allows layers to update in a purely feed-forward manner.
To further boost efficiency, Xiaohan Zhao introduced AsyncFGD, which uses a module-wise asynchronous pipeline. In standard training, the hardware sits idle while waiting for the backward pass to reach specific layers. AsyncFGD allows different modules to process different data samples simultaneously in a forward-only flow, maximizing hardware utilization while keeping the memory footprint at near-inference levels.
Compressing the Breadcrumbs
When backpropagation cannot be avoided entirely, researchers focus on shrinking the "breadcrumbs" (activations) that must be stored. Le-Trung Nguyen and colleagues proposed Activation Map Compression through Tensor Decomposition, which uses Higher-Order Singular Value Decomposition (HOSVD) to compress activation maps by up to 18x.
Similarly, Hung-Yueh Chiang developed MobileTL, which approximates activation gradients using binary masks. Instead of storing full-precision 32-bit tensors for every layer, the system stores a much smaller bitmask that captures the essential information needed to reconstruct the gradient. This allows complex architectures like MobileNet to be fine-tuned on devices as small as a Raspberry Pi.
Why This Matters
The shift toward forward-only and memory-efficient learning is about more than just saving RAM; it is about autonomy. By removing the need for massive "memory banks" of activations, these techniques allow Large Language Models (LLMs) and speech foundation models to adapt to a user's specific voice or vocabulary directly on their device, without ever sending sensitive data to the cloud. As these methods mature, the distinction between "inference-only" edge hardware and "training" hardware will continue to blur, enabling a new generation of truly adaptive, private, and efficient AI.