High-Rank Test-Time Adapters on a Forward-Only Budget

Models deployed on drones, phones and embedded boards meet conditions their training data never covered: night instead of day, satellite views instead of street photos, sensor noise of a new kind. Test-time adaptation, which means updating a model on the unlabeled data stream it sees after deployment, is the standard response. On edge hardware, though, the usual tool of backpropagation is often out of reach, because it must store every intermediate activation of the network. The cheapest workaround is to optimize a small adapter using only forward passes. Zeroth-order methods do this by nudging the parameters randomly and watching how the loss changes. Their weakness is that the noise in this gradient estimate grows with the number of parameters being nudged. As a result, every forward-only method so far uses a tiny, low-rank set of adjustable parameters. That design runs straight into representation rigidity: when the frozen backbone lacks the features a new domain needs, small low-rank corrections cannot supply them.

This note proposes separating two quantities that current designs treat as one: how many parameters must be perturbed, and how expressive the resulting weight update is. A Hadamard-factorized adapter builds its update as the element-wise product of two low-rank matrices. It can reach rank $r^2$ while keeping the parameter count of a rank-$r$ LoRA adapter. Zeroth-order noise depends on parameter count, not rank, so this extra expressivity should cost nothing in gradient-estimate quality, memory, or forward passes. We first measure whether test-time updates actually need high rank, and under which kinds of shift. We then test whether the free extra rank buys accuracy exactly where the frozen representation falls short.

Core Claim

Under a fixed forward-pass budget, the right figure of merit for a test-time adapter is expressivity per perturbed parameter, not parameter count alone. We predict two things. On low-level corruptions, the best supervised update for an adapter layer has low effective rank, so rank-$r$ LoRA and Hadamard adapters will tie. On semantic and sensor-level shifts, such as renditions, remote sensing and night driving, the best update has markedly higher effective rank. There, a Hadamard adapter optimized with zeroth-order steps will beat a LoRA adapter with the same parameter count under the same optimizer, with no extra memory or forward passes. If the first prediction fails everywhere, meaning test-time updates are low-rank even under semantic shift, the proposal turns into a clean negative result: rank is not the edge bottleneck, and effort should go elsewhere.

Where This Sits

Two bodies of work pull in opposite directions, and this note is built on that tension.

On one side, ViDA argues that test-time adaptation needs a high-rank adapter branch (bottleneck width 128) to capture domain-specific shifts. ABBA-Adapters shows that LoRA's rank cap limits what offline fine-tuning can learn, and lifts it with a Hadamard product of two low-rank pairs. Both depend on backpropagation. ABBA was only tested in supervised, offline language-model tuning.

On the other side, CAZO and PACE motivate the low-rank assumption. CAZO observes that the curvature of the test-time loss is low-rank (its top 20 eigenvalues carry over 96% of the variance) and runs zeroth-order steps on an early-layer adapter. PACE finds that 500 to 2000 subspace components suffice for normalization-layer updates searched by an evolution strategy. Both were measured mainly on ImageNet-C. The Many Faces of Robustness warns that methods tuned to ImageNet-C often fail on semantic shifts like ImageNet-R. So their low-rank finding may describe corruptions, not distribution shift in general.

Spectral methods sit in between. IMSE tunes only the singular values of pretrained layers. BOLT tunes a diagonal over a basis extracted from a library of fine-tuned models. Both are compact, but both still backpropagate, and both cannot leave the span of their frozen bases.

The rigidity literature shows this is not a hypothetical concern. Studies on pathology CLIP adaptation, nighttime UAV tracking, and medical few-shot adaptation report that lightweight modules hit a ceiling when the backbone lacks the relevant features. The medical study adds that complex prompt tuning often fails to beat a linear probe. That result constrains us: more machinery is not automatically better, and a simple head must stay in the comparison.

The method pieces are reused, not invented. We take ABBA's factorization and its scaling, and CAZO's curvature-aware zeroth-order optimizer. What is new is the joint claim and the test: rank and perturbed dimension come apart under forward-only optimization, and this matters specifically where the representation is rigid.

Research Program

Diagnostic stage. For each shift type, we compute an oracle update offline. This is a supervised, backpropagated, unconstrained update to the weights of a candidate adapter layer, using target labels. We record its singular-value spectrum and effective rank, measured as the exponentiated entropy of its normalized singular values. We repeat this at an early, middle, and late block of a ViT-Base. The shifts are:

  • ImageNet-C corruptions,
  • ImageNet-R renditions,
  • the 15-dataset remote-sensing suite used by BOLT,
  • the night and fog splits of Cityscapes-to-ACDC.

This stage tests CAZO's low-rank claim outside corruptions. It also checks CAZO's early-layer placement against the expectation that semantic gaps live deeper in the network. A second offline measurement asks how much of each oracle update lies outside the span of the pretrained singular vectors. That fraction is exactly the part IMSE-style methods cannot reach.

Method stage. We insert a Hadamard adapter at the layers chosen by the diagnostic. Its weight update is

$$\Delta W = s\,(B_1A_1)\odot(B_2A_2),$$

where $B_i \in \mathbb{R}^{d\times r}$ and $A_i \in \mathbb{R}^{r\times k}$ are low-rank factors, $\odot$ is element-wise multiplication, and $s \propto 1/\sqrt{r_1 r_2}$ is ABBA's stability scale. The update can reach rank $r^2$ with $2r(d+k)$ parameters.

Following ABBA, we initialize $(B_1,A_1)$ from a truncated decomposition of the frozen weight and set $B_2=0$. The update therefore starts at zero and initially acts as a learned element-wise modulation of the pretrained weight. Unlike IMSE's singular-value rescaling, this modulation is not confined to pretrained singular directions.

All parameters are updated with CAZO's curvature-aware two-point estimator on an unsupervised entropy objective. We use the same number of forward perturbations per step that CAZO uses. The backbone stays frozen and can be quantized to INT8, since zeroth-order steps never differentiate through it.

The known risk is multiplicative coupling. With $B_2=0$, the first factor pair receives zero gradient signal at the start. The two pairs may also trade scale in ways that make random perturbation inefficient. We will test perturbing the two pairs on alternating steps, and report it as a design choice rather than tune it silently.

Placement stage. We cross adapter type (LoRA or Hadamard, matched in parameter count) with insertion depth (early, middle, late) for each shift family. If the expressivity advantage appears only for deep placement on semantic shifts, that supports a rigidity story. Forward-only alignment of early features cannot reach that regime.

Baselines

Unadapted model. The reference point is the frozen source ViT-Base. NEO reports 55.6% ImageNet-C accuracy for this model (NEO).

Strongest existing forward-only method: CAZO. CAZO runs zeroth-order optimization of a LoRA-style early adapter. It reports 69.0% average ImageNet-C accuracy, against 59.8% for TENT and 65.8% for FOA. It cuts memory from 6,404 MB (TENT) to 1,695 MB (CAZO). It is the most direct competitor because it shares our optimizer and memory profile and differs only in adapter structure. We rerun it with the authors' settings on our streams, using:

  • the same backbone and stream order,
  • the same number of forward passes per sample,
  • the same tuning budget for learning rate and perturbation scale, searched on held-out corruptions only.

Beating it means higher accuracy at equal forward passes and peak memory on ImageNet-R, remote sensing, and ACDC night. On ImageNet-C we expect parity, not a win, and parity there is a prediction of the hypothesis.

Key ablation: matched-parameter LoRA. The ablation isolating our ingredient is CAZO's optimizer applied to a rank-$2r$ LoRA adapter, which has exactly the Hadamard adapter's parameter count. We also run rank-$r$ LoRA at lower cost. This separates "more rank" from "more parameters." An ABBA adapter trained with backpropagation gives an upper bound on what the structure can reach without zeroth-order noise.

Forward-only alignment, with no learned weights. We compare against PEA, which aligns embedding statistics with a whitening-coloring transform. It reports 64.5% on ImageNet-C using 887 MB and 0.31 s per batch. We also compare against NEO's running-mean re-centering, which reports 59.2% at 64 samples. These show whether learned updates are worth their forward-pass cost at all.

Derivative-free normalization tuning. PACE reports 65.0% on ImageNet-C with an 8-bit model and 13 FPS on a Jetson Xavier NX. It represents the "tune only what already exists" position.

Backpropagated high-rank adapter. ViDA reports 61.9% mIoU on Cityscapes-to-ACDC. It serves as the accuracy ceiling for backprop-heavy methods and is run where memory allows.

Gradient-detached bypass. A Res-Tuning-Bypass branch is trained with exact gradients on the same entropy objective. Gradients flow only through the bypass.

For remote sensing, the label-free reference is BOLT's 67.74% against 56.87% for zero-shot CLIP on ViT-B/32. No reported result was found on Lacuna for forward-only methods on this suite. No reported result was found on Lacuna for any test-time method's ImageNet-R accuracy, so those numbers will come from our reruns.

Why not just…?

Why not just raise the LoRA rank under zeroth-order optimization? Doing so raises the number of perturbed parameters. Zeroth-order gradient noise grows with that number, so extra rank is paid for in noise or in forward passes. The matched-parameter rank-$2r$ LoRA ablation is the direct test. If it matches the Hadamard adapter on semantic shifts, our structural claim fails, and we will report that.

Why not just use forward-only feature alignment like PEA or NEO? These methods are faster and need no optimization. On ImageNet-C they trail CAZO (64.5% and 59.2% against 69.0%). They can only translate, scale, and rotate features the backbone already produces, which is the regime where rigidity says they should stall. This is a real threat on corruptions. The decisive evidence is whether their gap to learned adapters widens on ImageNet-R, remote sensing, and night driving compared with ImageNet-C. If it does not widen, alignment is the better edge choice.

Why not just backpropagate through a detached bypass branch? This is the most serious alternative. A detached bypass gets exact gradients while storing only its own activations, and Res-Tuning reports 49.7% lower training memory for its Bypass variant. Its limit is that it reads frozen backbone features and cannot change intermediate representations. It is also untested in unsupervised streaming adaptation. We run it at matched memory. If it equals the Hadamard adapter on semantic shifts, the case for zeroth-order in-backbone updates reduces to settings without any autodiff runtime, as noted by Patacchiola et al.. We will state that scope plainly.

Why not just a linear probe or classifier-only adaptation? A probe-style method such as T3A uses only 1.05x inference memory on a Raspberry Pi 4B (BoTTA), and the medical results above show such heads are hard to beat. CAZO's ablation found intermediate adapters outperform final-layer tuning on corruptions. Our placement stage tests whether that still holds for semantic shifts.

Decisive Checks

All methods use ViT-Base and continual streams over ImageNet-C (severity 5), ImageNet-R, the remote-sensing suite, and Cityscapes-to-ACDC. We also run BoTTA's harsher protocol: batches under 64 samples and overlapping corruptions. BoTTA reports SHOT falling from 79.9% to 39.1% under three overlapping corruptions, so this protocol stresses stability.

We report accuracy or mIoU, peak memory, and latency per batch on a Jetson Orin Nano with 3.5 GB of RAM. We also report forward passes per sample, and accuracy on an INT8 backbone.

The project succeeds if three things hold together:

  1. The diagnostic shows clearly higher oracle effective rank on semantic and sensor shifts than on corruptions.
  2. The Hadamard adapter beats matched-parameter LoRA-ZO and CAZO on those shifts at identical forward-pass and memory budgets, while tying on ImageNet-C.
  3. The gain over the detached bypass and PEA grows with the measured oracle rank.

Other outcomes are informative in different ways. If oracle rank is low everywhere, the low-rank premise of edge test-time adaptation is confirmed beyond corruptions. If the rank is high but zeroth-order steps cannot exploit the Hadamard structure, the coupling problem is the next thing to solve. Both outcomes would settle the open contradiction between ViDA and CAZO.