Low-Rank Feature Alignment at the Cut Layer for On-Device Continual Learning

Small devices that learn after deployment, such as a microcontroller-powered camera or a sensor node, usually split their network in two. The early layers (the "front end") stay frozen. Only the later layers are trained, using a small buffer of stored intermediate features from earlier data, called latent replay, to avoid forgetting old classes. This design works because the front end never changes. That is also its weakness. When the environment shifts (night falls, fog rolls in, the lens degrades), the frozen front end produces distorted features. Those features no longer live in the same coordinate system as the replay buffer, so the trained back end must reconcile two incompatible inputs. We propose a closed-form fix with no backward pass. From a handful of unlabeled images, estimate the dominant directions of the feature distribution at the split point with a truncated singular value decomposition (SVD), the standard factorization that finds a matrix's most important directions. Then apply a single affine map that rotates and rescales the new features back into the frame the replay buffer was recorded in.

The bet is not simply that "SVD alignment helps." Cheaper and better-known fixes exist at the two ends of a spectrum. At one end, matching per-channel means and variances is cheap but ignores interactions between channels. At the other, full-covariance matching captures everything but cannot be estimated from the few dozen images an edge device actually sees. A rank-$k$ subspace sits between them, and the rank works as a shrinkage knob. Our claim is that an intermediate rank, chosen from the number of available samples, beats both ends exactly in the regime where on-device test-time adaptation is known to fail: fewer than 64 adaptation samples and overlapping corruptions.

Core Claim

At the cut layer of a latent-replay network, a rank-$k$ subspace alignment estimated in closed form from $n \le 64$ unlabeled images should beat three things on the post-shift accuracy of a continually learning device: per-channel statistic matching, full-covariance matching, and gradient-based test-time adaptation. Because the alignment is affine at each spatial position, it folds into the weights of the first trainable layer. Inference therefore costs nothing extra, and the stored INT8 replays remain valid. The claim is falsifiable. If per-channel matching ties the low-rank map across sample counts and corruption mixes, the subspace adds nothing and we will report that.

The map is the following. Let $z \in \mathbb{R}^C$ be a $C$-channel feature vector at one spatial location of the cut layer. Let $\mu_S, \mu_T$ be the source and target means. Let $X_S \in \mathbb{R}^{C \times K}$ be the stored source basis (top-$K$ singular vectors from deployment-time data), and $X_T \in \mathbb{R}^{C \times k}$ the target basis estimated on-device with $k \le K$. Then

$$z' = \mu_S + X_S X_S^{\top} X_T \Lambda X_T^{\top}(z - \mu_T) + \beta\,(I - X_T X_T^{\top})(z - \mu_T),$$

where the $k \times k$ diagonal matrix $\Lambda$ rescales each target direction to the source standard deviation along that same direction, and $\beta \in \{0, 1\}$ either discards or keeps the off-subspace residual.

The middle term is the subspace-alignment map of Unsupervised Visual Domain Adaptation Using Subspace Alignment, $M = X_S^\top X_T$, lifted back to the full feature space. Choosing the coordinate axes as the basis recovers per-channel matching, and $k = C$ recovers full-covariance matching. Both competitors are therefore points on one curve, and the experiment asks where on the curve the optimum lies.

Position Relative to Prior Work

Three papers motivate the problem. Lightweight Unsupervised Domain Adaptation by Convolutional Filter Reconstruction shows that lighting-type shifts distort responses already at the first convolutional layer. It repairs them by Lasso-reconstructing unstable filters from 1–10 unlabeled images. This is evidence that statistics-based, closed-form repair works where gradients are unavailable, but the method touches only layer one and does not consider continual learning. A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays and Structured Sparse Back-propagation for Lightweight On-Device Continual Learning on Microcontroller Units freeze everything up to the cut layer and assume that the stored latents stay meaningful. Neither tests what happens when the input distribution shifts mid-stream, which is the gap we target. BoTTA: Benchmarking on-device Test Time Adaptation supplies the failure regime: gradient-based methods fail to improve the source model below 64 samples and collapse under compound corruptions.

Two papers supply the method. Subspace Alignment ( Unsupervised Visual Domain Adaptation Using Subspace Alignment) gives the closed-form map but runs offline on pre-extracted features, with a fixed dimension and no stream or replay. Test-time Adaptation for Regression by Subspace Alignment shows that restricting alignment to a low-dimensional, output-relevant subspace stabilizes it against nonlinear distortions. However, it still runs iterative KL-divergence gradients. We keep its insight and drop the gradients.

Several papers constrain the design. Continual Learning in Low-rank Orthogonal Subspaces needs oracle task identifiers and Stiefel-manifold updates with matrix inversions, so we avoid per-task subspaces entirely. Orthogonal Projection Subspace to Aggregate Online Prior-knowledge for Continual Test-time Adaptation handles continual weather shifts with soft-orthogonal adapters and a student–teacher pair. That approach needs GPU backpropagation, which rules it out on microcontrollers. We borrow only its trick of merging the adaptation into existing weights so that inference cost stays flat. The weight-space SVD methods DisLoRA and Dual Decomposition of Weights and Singular Value Low Rank Adaptation factor pretrained weights for language-model fine-tuning. Their own description notes that full SVD of large matrices is costly. We avoid this cost by factoring a small sample-by-channel activation matrix with a randomized thin SVD, never a weight matrix.

Research Program

The system has three parts. At deployment, the device stores the source mean $\mu_S$ and an INT8 basis $X_S$ with $K$ around 64 together with its singular values. For a 256-channel cut layer this costs roughly 16 KB.

When a shift is detected or scheduled, the device collects $n$ unlabeled images and forms the matrix of cut-layer vectors. There is one row per spatial location, so ten images at 8×8 resolution already give 640 rows. This is why the estimate is feasible from few images. The caveat is that neighboring locations are correlated, so the effective sample count is smaller than the row count. The device then runs a randomized range-finder SVD with a few power iterations, at cost roughly proportional to rows × $C$ × $k$. It computes $\Lambda$ and folds the resulting affine map into the first trainable layer's weights and bias.

Training of the back end on new classes then proceeds as in the latent-replay platforms, against an unchanged INT8 replay buffer.

The scientific core is the choice of rank. The number of free parameters in a rank-$k$ basis grows as roughly $kC$. Subspace estimation error should therefore grow with $kC$ divided by the effective sample count. This predicts that the best $k$ increases with $n$. We will test that prediction directly and fit a simple rule, $k$ as a function of $n$, that the device can apply without labels. We will also measure whether discarding the residual ($\beta = 0$) helps under compound corruptions, as the noise-filtering argument suggests, or hurts under clean conditions by throwing away discriminative low-variance directions. A secondary measurement asks whether storing replays as $k$-dimensional coordinates in $X_S$ shrinks the buffer beyond INT8 without accuracy loss. We report it but do not treat it as the main claim.

Baselines

The naive reference is the source model with the frozen front end and latent replay, with no alignment. For the continual setting we reproduce the quantized latent replay pipeline of A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays, which reports a 0.26% accuracy cost from INT8 replays. For the CORe50 new-classes scenario we use the anchors in CORe50: CWR at 35–40% and a cumulative upper bound of about 75%.

The most dangerous competitor is per-channel statistic re-estimation at the cut layer. This means recomputing batch-normalization or channel mean/variance on the target images, the diagonal endpoint of our curve. It costs almost nothing and requires no factorization. We found no reported result on Lacuna for per-channel statistic re-estimation on CORe50 or the BoTTA corruptions, so we will run it ourselves. It gets the same $n$ images, the same cut layer, and the same folding into weights. The other endpoint is full-covariance alignment ($k = C$, with shrinkage tuned on a held-out source-side shift). Beating both endpoints means a statistically significant gain, across seeds, of the sample-count-chosen $k$ at $n \le 64$ on compound corruptions.

Among gradient-based test-time adaptation methods, we run SHOT, TENT and T3A under the BoTTA protocol on a Raspberry Pi 4B. BoTTA reports SHOT at 79.9% accuracy with a single corruption and 39.1% with three overlapping corruptions, with peak memory 1.08× inference for SHOT and 1.05× for T3A. We will give them the same adaptation images and the BoTTA hyperparameters. Beating SHOT means higher accuracy in the three-corruption, under-64-sample cell at comparable or lower peak memory.

For first-layer repair we reimplement filter reconstruction ( Lightweight Unsupervised Domain Adaptation by Convolutional Filter Reconstruction) with the same sample budget. For "just unfreeze more," we run structured sparse backpropagation through the front end following Structured Sparse Back-propagation for Lightweight On-Device Continual Learning on Microcontroller Units, which reports a 2% accuracy drop versus full training and 4.63 MB of RAM for MobileNetV2. To isolate the key ingredient, ablations sweep $k$ continuously, toggle $\Lambda$ and $\beta$, and move the alignment from the cut layer to the first layer.

Why not just…?

Why not just re-estimate per-channel normalization statistics? This is the real threat, and the experiment is built around it. If shifts at the cut layer were mostly per-channel gain and offset changes, diagonal matching would capture them, and the optimal $k$ in our sweep would collapse to the axis-aligned endpoint. The filter-reconstruction paper's finding that distorted filters must be rebuilt as combinations of other filters suggests that the shift mixes channels. Compound corruptions should mix them further. We will settle this empirically by plotting accuracy against $k$ with the diagonal baseline as a reference line.

Why not just run gradient test-time adaptation like SHOT or TENT? BoTTA shows these methods fail below 64 samples and lose more than half their accuracy under three overlapping corruptions. They also require backward passes that microcontrollers struggle to afford. Gradient methods may still win when thousands of samples are available; BoTTA reports SHOT's 2.74× gain at 8,192 samples. We will report the sample count at which they overtake closed-form alignment rather than claim uniform superiority.

Why not just retrain the classifier head on new data? A head trained on distorted features must relearn the shift for every class. The replay buffer, recorded in the old frame, then actively conflicts with the new data. Our continual experiment measures this directly as forgetting of old classes after a mid-stream shift, with and without alignment.

Why not just use full-covariance alignment? With $C = 256$ channels, a full covariance has about 33,000 free entries. A few dozen images cannot estimate it without heavy shrinkage. In addition, storing and inverting it on a microcontroller costs far more than a rank-32 basis. Full covariance is our other endpoint, so the rank sweep will show whether shrinkage alone closes the gap.

Proposed Experiments and Decisive Checks

The main testbed is CORe50 with MobileNet-V1 and MobileNetV2 latent-replay cut layers, matching the two latent-replay platforms. We use the new-instances and new-instances-and-classes scenarios. The device is trained on indoor sessions and then switched mid-stream to the outdoor sessions, a natural shift. We additionally inject single and stacked corruptions in the style of the BoTTA protocol. Adaptation budgets range from 1 to 64 unlabeled images. Metrics are average accuracy over the stream, forgetting of pre-shift classes, and peak RAM.

The second testbed is the BoTTA corruption suite on a Raspberry Pi 4B, for direct comparison with SHOT, TENT and T3A. The third is the 27-week IoT device identification data of Revisiting IoT Device Identification, where accuracy decays by 17% on average from behavioral drift. It is a non-vision check of whether cut-layer alignment of a small traffic classifier recovers part of that decay. We will measure on-device cost of the estimation step on an ARM Cortex-M4 with INT8 storage of the basis, reporting latency, energy and RAM against a single training step of the back end.

Three outcomes decide the proposal. First, in the compound-corruption, $n \le 64$ cells, the best intermediate $k$ must beat both diagonal and full-covariance matching. Second, the optimal $k$ must increase with $n$ as the estimation-error argument predicts. If it does not, the sample-count rule is unfounded and $k$ becomes a tuned hyperparameter. Third, after a shift on CORe50, forgetting of old classes with alignment must fall relative to the unaligned latent-replay pipeline, with the replay buffer unchanged. If the first check fails, the paper becomes a negative result: diagonal matching suffices at the cut layer, which is itself useful guidance for microcontroller continual learning.