When we train a deep learning model in a cloud data center, we take massive hardware resources for granted. Standard backpropagation requires holding the entire model architecture in memory, alongside forward activations for every layer, computed gradients, and auxiliary optimizer states (such as the running averages in Adam, which can triple the parameter storage requirements).
On an edge device—such as a smartphone, drone, or IoT sensor—this workflow hits a hard wall. As noted by Hongchang Gao et al., high-dimensional models make standard backpropagation impossible on mobile hardware due to severe RAM and computational limits. This creates a fundamental mismatch when trying to deploy modern foundation models or collaborative frameworks like Federated Learning (FL), where data privacy principles dictate that training must happen locally on-device (Katharine Daly et al.; Chunxu Zhang et al.).
To bridge this gap, researchers have developed three core strategies that fundamentally alter how networks are updated: shrinking the parameter update space, partitioning the model architecture across entities, and optimizing local memory and data management.
Shrinking the Update Space via Parameter-Efficient Tuning
The most direct way to bypass edge memory constraints is to avoid updating the entire model. If the vast majority of a network's weights remain frozen, the device does not need to store optimizer states or track gradients for those parameters, drastically lowering the memory ceiling.
Parameter-Efficient Tuning (PETuning) adapts this concept for decentralized environments. Zhuo Zhang 0007 et al. demonstrate in FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models that updating only a tiny fraction of a model's parameters can achieve over 95% of the performance of full fine-tuning while slashing communication and local compute overhead.
To push this efficiency further, Ahmed Roushdy Elkordy et al. introduce SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models, which uses a sparse-to-low-rank transition to minimize update density down to 1%. Similarly, Haris Vikalo et al. leverage dynamic sparse training in Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better to let parameter masks evolve dynamically during training, ensuring edge devices only compute a fraction of the weights at any given step.
Standard Fine-Tuning:
[Compute Gradients & Optimizer States for All Parameters] -> Overwhelms Edge RAM
Parameter-Efficient Tuning (e.g., LoRA / Adapters):
[Frozen Backbone] -> [Tiny Trainable Adapter] -> Low Memory Footprint
For vision and speech models, freezing the backbone and training small auxiliary modules provides a lightweight path to personalization. Stefanos Laskaridis proposes attaching and training intermediate "early exits" rather than the full network backbone in It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation, which drastically reduces the backward pass overhead. For text classification, Ali Shakeri utilizes a tiny auxiliary network to generate context-aware prompts dynamically, bypassing the need to modify the underlying large language model entirely.
Architecture Partitioning and Split Learning
When a model is too large to fit into a single device's memory even when frozen, the architecture itself must be broken apart. Split Learning (SL) cuts the neural network into segments, distributing the computational layers between the resource-constrained client and a powerful central server.
In a typical split configuration, the edge client hosts and trains only the initial layers. It processes the input data, computes the intermediate activations (the "smash data"), and passes them to the server, which executes the remaining layers and handles the bulk of the backpropagation. Chandra Thapa et al. formalize this hybrid approach in SplitFed: When Federated Learning Meets Split Learning, combining the parallel aggregation efficiency of FL with the model-splitting benefits of SL to allow low-power IoT devices to participate in large-scale training.
Edge Client (Low Power): [Input Data] -> [Initial Layers] -> (Activations) ---\
|--> Network Transfer
Central Server (High Power): [Final Layers] <--/
This partitioning can adapt dynamically to handle hardware heterogeneity across devices. Seyed Mahmoud Sajjadi Mohammadabadi et al. propose a dynamic tiering approach where slower, more resource-constrained devices offload a larger portion of their computational layers to the server, preventing stragglers from delaying the global training loop.
To scale this to complex multimodal architectures, Zixin Zhang et al. combine U-shaped split learning with lightweight adapters in Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models, allowing edge nodes to contribute to multimodal models without storing billions of parameters. Even for generative models, Corentin Hardy et al. show in MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets that hosting the generator centrally while distributing lightweight discriminators to local workers cuts the memory burden on edge nodes by half.
Algorithmic and Data-Level Memory Management
Beyond changing what parameters are updated or where they reside, on-device training can be optimized by re-engineering the underlying mechanics of backpropagation and data storage.
At the operator level, memory can be reclaimed through clever scheduling. Jianxin Zhao et al. introduce Operator-Level Parallel Backpropagation in Parallel and Memory-Efficient Distributed Edge Learning in B5G IoT Networks. By breaking down layers into individual mathematical operators and applying a "pebble-game" heuristic for memory management, they optimize the reuse of memory buffers, significantly cutting the local RAM footprint.
On the optimization side, advanced tracking algorithms often require storing auxiliary variables that scale with model size. To combat this, Xin Zhang et al. design GT-STORM: Taming Sample, Communication, and Memory Complexities in Decentralized Non-Convex Learning, a single-loop decentralized optimization algorithm that cuts memory usage in half by fusing gradient tracking with stochastic recursive momentum, eliminating redundant tracking variables.
Numerical precision offers another lever for optimization. Bokun Wang et al. demonstrate in Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point that executing both local training and client-server communication in 8-bit floating point (FP8) precision drastically minimizes memory usage while maintaining model accuracy.
Standard Precision (FP32): [4 Bytes per Weight/Activation] -> High Memory Overhead
Low Precision (FP8): [1 Byte per Weight/Activation] -> 4x Memory Reduction
Finally, memory constraints apply equally to data storage. Edge devices cannot store massive historical datasets for continual or self-supervised learning without running out of disk space. Chen Gong et al. address this in To Store or Not? Online Data Selection for Federated Learning with Limited Storage via an online data selection framework that selectively caches only the streaming samples that align closely with the global gradient.
For self-supervised learning, which typically demands massive batches to remain stable, Yawen Wu et al. introduce a contrast-score filtering mechanism in Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast. This maintains a tiny, highly informative data buffer, making on-device contrastive learning feasible without overwhelming local flash memory.