Deep neural networks deployed in the real world inevitably face distribution shifts—a drone flying into a sudden snowstorm, or an autonomous vehicle navigating a foggy night. Test-Time Adaptation (TTA) offers an elegant solution: allow the model to update its parameters on the fly using unlabeled streaming data from the new environment.
However, deploying standard TTA on edge devices reveals a harsh reality. The very mechanism that enables learning—backpropagation—is fundamentally mismatched with the hardware constraints of edge devices, microcontrollers, and mobile sensors. A growing body of research focuses on diagnosing and eliminating this backpropagation overhead to make real-time, on-device adaptation practical.
The Hidden Cost of Gradients
To understand why backpropagation fails on the edge, we must look at how memory is managed during inference versus training.
During a standard inference forward pass, data flows sequentially from layer to layer. Once a layer's activations are used to compute the next layer's inputs, they are no longer needed and can be immediately purged from memory. The peak memory footprint of inference is therefore bounded roughly by the size of the single largest layer.
Inference (Forward Pass Only):
[Layer 1] ---> (Activations 1) ---> [Layer 2] ---> (Activations 2) ---> [Layer 3]
[Purged] [Purged]
When we introduce backpropagation for online adaptation, this ephemeral memory model breaks. To compute the gradient of the loss with respect to a layer's weights, the chain rule requires access to that layer's original forward activations. Consequently, the device must retain the intermediate activations for every single layer across the entire network graph until the backward pass completes.
Adaptation (Forward + Backward Pass):
[Layer 1] ---> (Stored Act. 1) ---> [Layer 2] ---> (Stored Act. 2) ---> [Layer 3]
| | |
[Grad 1] <----------------------------- [Grad 2] <--------------------------- Loss
As highlighted by Song and Deng et al., this activation memory bottleneck scales with network depth and input resolution, making standard TTA completely impractical for memory-constrained edge hardware like smartphones or IoT sensors.
Beyond memory capacity, the computational and energy costs of calculating gradients pose severe challenges. Ankit et al. note that full-parameter updates cause massive hardware-level inefficiencies in memory access and multiplication-accumulation (MAC) operations. Furthermore, many edge platforms lack native hardware or software support for automatic differentiation entirely, making even lightweight gradient updates computationally prohibitive (Patacchiola et al.). For mobile robots or drones, this computational overhead translates directly into shortened battery life and real-time latency bottlenecks (Bao et al.; Rifat et al.).
To circumvent these constraints, researchers have developed several distinct paradigms that eliminate or minimize backpropagation during test-time adaptation.
Paradigm 1: Backpropagation-Free Alignment
The most direct way to solve the activation memory bottleneck is to eliminate the backward pass entirely. Instead of optimizing weights via gradients, "backward-free" adaptation methods realign shifted distributions using only forward passes.
For instance, Ma et al. introduced Progressive Embedding Alignment (PEA). PEA geometrically realigns intermediate feature embeddings using just two forward passes, completely bypassing gradient computation while remaining architecture-agnostic.
In semantic segmentation, where high-resolution images exacerbate activation storage costs, Wang et al. proposed Dynamically Instance-Guided Adaptation (DIGA). DIGA replaces optimization-based updates with non-parametric distribution and semantic modules, achieving state-of-the-art accuracy without backpropagation.
Similarly, for 3D datasets, Wang et al. developed BFTT3D, which pairs a frozen source model with a non-parametric adaptation stream to align features in a shared subspace. By avoiding noisy pseudo-labeling and backpropagation, these approaches drastically cut memory and compute costs. This philosophy is mirrored by the authors of NEO, who leverage latent re-centering to achieve test-time adaptation entirely without optimization.
Paradigm 2: Zeroth-Order Optimization
What if an application still requires optimizing a loss function, but the underlying model cannot support backpropagation? This is a common issue in Quantized Neural Networks (QNNs), where low-bit weights are non-differentiable, rendering standard gradient descent useless.
Zeroth-order optimization provides an elegant workaround by estimating gradients using only forward passes. Deng et al. developed Zeroth-Order Adaptation (ZOA) to solve this problem.
$$\nabla f(w) \approx \frac{f(w + \mu u) - f(w)}{\mu} u$$
By perturbing the weights slightly with a random vector $u$ and evaluating the objective function via two forward passes, ZOA estimates a descent direction without ever computing a partial derivative. Because it eliminates the need to store intermediate activations, ZOA allows low-bit quantized models to adapt to new domains within a minimal memory footprint.
Paradigm 3: Parameter-Efficient and Hybrid Architectures
Rather than modifying how gradients are computed, another school of thought limits where updates occur. By isolating adaptation to a tiny fraction of the model's parameters, systems can drastically reduce the scale of the backward pass.
Wei et al. introduced CoPEFT, a framework designed for multi-agent collaborative perception in autonomous vehicle fleets. CoPEFT updates less than 1% of a model's parameters during fine-tuning, allowing rapid environmental adaptation without introducing high latency into real-time perception loops.
Taking this a step further, hybrid architectures replace complex deep learning heads with highly efficient classical solvers that can be updated analytically. Ceola et al. demonstrated this by replacing the heavy, backpropagation-based heads of standard segmentation models with fast, kernel-based classifiers (FALKON). This allowed the iCub humanoid robot to learn and segment new objects on-the-fly without extensive downtime or heavy GPU resources.
Similarly, Patacchiola et al. combined Contextual Squeeze-and-Excitation (CaSE) blocks with a lightweight linear head, pushing the Pareto frontier of on-device personalization by avoiding full network fine-tuning.
Paradigm 4: Structural Pruning and Statistics Management
A final strategy focuses on optimizing the structural and logical overhead of the adaptation process itself.
During continuous adaptation, many feature channels may become redundant or react poorly to environmental shifts. Wang et al. developed a sensitivity-guided pruning framework for object detection that identifies and suppresses these "domain-sensitive" channels. By pruning these channels dynamically, the system reduces the total number of floating-point operations (FLOPs) required during both the forward pass and parameter updates.
Concurrently, researchers have focused on minimizing the logical overhead of tracking data streams. The authors of Bag of Tricks for Fully Test-Time Adaptation demonstrated that carefully optimizing normalization layers and managing class-frequency statistics can make TTA robust even down to a batch size of 1. This eliminates the need to buffer large batches of data on-device, mitigating the memory and logic overheads that complicate edge deployment.
Key Foundational Work
Deep Structured Features for Semantic Segmentation (2017) — contributed 1 concept elements to this direction.
Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner (2020) — contributed 1 concept elements to this direction.
Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing (2020) — contributed 1 concept elements to this direction.