Test-Time Adaptation as a Local Gaussian Process Correction to the Source Classifier

When an image classifier is deployed on data that differs from its training data (blurred, snowy, sketched, or seen through a different camera), its accuracy drops. Test-time adaptation, which updates the model on the unlabeled stream of deployment inputs, is the standard remedy. Current methods share a structural weakness. They change the classifier everywhere at once, either by nudging normalization parameters with gradient steps or by replacing the classifier head with averages of confident test features. A wrong early guess therefore spreads across the whole input space, and adapting to one shift overwrites what worked for the previous one. We propose to keep the source classifier as the default and learn only a local correction to its outputs. That correction is a Gaussian process, a probabilistic model of functions that also reports how confident it is at each point. It is fitted to pseudo-labels from the test stream, and its correction fades back to zero wherever the stream has provided little or inconsistent evidence.

The two-sentence version: the adapted classifier equals the frozen source classifier plus a correction term that is a Gaussian process over the network's feature space, fitted at test time from pseudo-labels. Because the Gaussian process's uncertainty grows away from observed evidence, the correction is automatically confined to regions where the stream actually supports it. This should resist both error amplification and forgetting, with no backpropagation and with the source model always recoverable. Prediction stays cheap because each query conditions only on its nearest stored neighbors, which avoids inverting a kernel matrix over the whole stream.

Core Claim

Residual adaptation, meaning learning a target model as "source plus a correction," has strong theory but has always needed target labels. Generalization Bounds for Transfer Learning under Model Shift proves that fitting an offset beats learning from scratch when the offset is smoother than the target function. Smoothness Adaptive Hypothesis Transfer Learning shows that two-phase kernel ridge regression with a tuned regularizer attains minimax rates for the offset. Active Transfer Learning under Model Shift models the offset as a Gaussian process and queries labels where its variance is high, but uses exact inference on fewer than a few hundred points. Transfer Learning with Kernel Methods scales an additive "translation" correction to millions of points with an iterative solver, but gives deterministic predictions and needs labels. All four motivate the residual form and supply its theory. None operates without labels.

The test-time literature is label-free but does not use a residual of this kind. Tent minimizes prediction entropy over normalization parameters, which changes the classifier globally. T3A replaces the linear head with centroids of confident test features, and its authors name simple averaging as a limitation. TAST uses nearest-neighbor consensus to choose pseudo-labels and trains small adapter modules on them. ADAPT and OGA fit parametric class Gaussians in closed form.

Our claim is that a source-anchored Gaussian process residual, fitted to pseudo-labels and evaluated locally, gives two things these methods lack. First, the adaptation is spatially confined, so updates do not leak to unrelated regions and old corrections persist where they remain valid. Second, there is a principled per-input measure of how much to trust the adaptation. We expect the payoff to be largest in continual, order-varying streams and in calibration, rather than in single-shift accuracy.

Mechanism

Let $z = \phi(x)$ be the frozen backbone's feature vector and $f_s(z) \in \mathbb{R}^C$ the source logits over $C$ classes. The adapted logits are

$$g(z) = f_s(z) + \Delta(z), \qquad \Delta \sim \mathcal{GP}(0, k),$$

with one shared kernel $k$ (cosine or RBF on normalized features) across the $C$ output channels. Each stored test feature $z_i$ contributes a pseudo-residual $r_i = t_i - f_s(z_i)$, where $t_i$ is a logit-space target built from a pseudo-label. To avoid self-reinforcement, the pseudo-label for point $i$ comes from the Gaussian process posterior computed excluding $i$, the same leave-one-out idea ADAPT uses for its class means. Each residual gets its own noise variance $\sigma_i^2$, which grows with the entropy of the leave-one-out prediction, so uncertain pseudo-labels pull on the correction less.

For a query $z_*$ we follow the decoupled nearest-neighbor scheme of GPnn: retrieve the $m$ nearest stored features by approximate search and compute the exact local posterior

$$\mu_\Delta(z_*) = k_*^\top (K_m + \Sigma_m)^{-1} R_m, \qquad s^2(z_*) = k(z_*,z_*) - k_*^\top (K_m + \Sigma_m)^{-1} k_*,$$

where $K_m$ is the neighbor kernel matrix, $\Sigma_m$ is the diagonal matrix of $\sigma_i^2$, $R_m$ stacks the $m$ residual vectors, and $k_*$ holds the query–neighbor kernel values. A single $m \times m$ solve serves all $C$ classes. Per-query cost is therefore $O(m^3 + m^2 C)$ regardless of stream length, which keeps 1,000-class ImageNet-C tractable where the authors of TAST report support-set memory trouble.

The prediction is $f_s(z_*) + \mu_\Delta(z_*)$. The variance $s^2$ does two jobs. It shrinks the correction toward the source wherever neighbors are far or noisy, which is implicit in the posterior mean. It is also reported as an adaptation-uncertainty score for selective prediction. The memory is a bounded first-in-first-out buffer per region, using reservoir sampling within approximate-search cells, so continual streams do not grow memory without limit.

Research Program

The first part of the work establishes the method on standard single-shift and domain-generalization benchmarks with a frozen backbone and source normalization statistics. The second part studies continual and order-varying streams, which is where locality should matter. The third part addresses three known tensions directly.

The first tension is dimensionality. GPnn concedes that nearest-neighbor search degrades above roughly 50 dimensions, while ResNet-50 features have 2048. We will compare search in the raw normalized features against a PCA projection to 64–256 dimensions computed from the first test batches. We will report retrieval recall alongside accuracy.

The second tension is hyperparameter choice. GPnn proves mean-squared error becomes insensitive to kernel misspecification as data grows, whereas SATL shows the offset's regularization must be tuned to reach good rates. In a source-free setting there is no labeled validation set. We will set the lengthscale and noise scale by maximizing local marginal likelihood on pseudo-residuals from the first few batches. We will then measure sensitivity across a tenfold range to learn which of the two results governs this regime.

The third tension is calibration. GPnn needs a labeled scalar recalibration step, which we cannot run. We will instead test whether $s^2$ ranks errors well enough to be useful even when its absolute scale is off.

Finally, we test composition with test-time normalization (The Norm Must Go On, MemBN). Composition is attractive because it repairs features cheaply. It is also risky, because stored features become stale as normalization statistics drift. We will compare refreshing stored features against discarding old ones.

Baselines

The naive reference point is the unadapted source model with frozen weights and source normalization statistics. Every method must beat it, and our method falls back to it by construction when the Gaussian process is uninformative. Next is test-time normalization, which recomputes normalization statistics on test batches. Tent reports 49.9% ImageNet-C error for it, and CoTTA reports 20.4% CIFAR-10-C continual error. It is the cheapest serious competitor and is orthogonal to our head-level correction, so we report both alone and combined.

Tent is the canonical gradient-based competitor: 44.0% error on ImageNet-C, and 20.7% in CoTTA's continual CIFAR-10-C protocol. T3A is the closest backpropagation-free head adjuster: +1.5 points over standard training across four domain-generalization benchmarks and 80.0% on VLCS. TAST is the strongest neighbor-based competitor and the one most likely to match us. The TAST paper reports 81.94% on PACS with ResNet-18, against 80.83% for T3A and 79.29% for standard training, and 13.08% CIFAR-10-C error at the highest severity for TAST-BN against 13.95% for Tent. In the continual setting the strongest method is CoTTA, at 16.2% CIFAR-10-C and 32.5% CIFAR-100-C error with full-network updates and 32 augmentations per step. On the CLIP natural-shift benchmarks, ADAPT reports 66.53% average online accuracy.

Fairness rules apply to all baselines. Every method uses the same backbones, the same stream order and batch sizes, and the same seeds. Every method gets the same number of unlabeled warm-up batches for hyperparameter selection, with no target labels used for tuning. Wall-clock time and peak memory are reported per method.

Beating TAST means higher PACS accuracy and higher accuracy averaged across all four domain-generalization benchmarks. Against CoTTA we do not expect to win on accuracy, since our backbone is frozen. Our goal there is to come within about one point of CoTTA's 16.2% at a small fraction of its compute, and to beat it on error averaged over repeated domain cycles and on worst-order runs.

Three ablations isolate the key ingredient. In the first, the correction is computed by plain kernel smoothing, giving the same neighbors and weights but no noise-aware posterior and no variance shrinkage; this is effectively a soft-kNN residual. In the second, the source prior mean is dropped by setting $f_s$ to zero, which turns the method into a Gaussian process classifier on pseudo-labels, close in spirit to T3A and TAST. In the third, leave-one-out pseudo-labels are replaced with the model's own current predictions.

Why not just…?

Why not just use TAST's nearest-neighbor consensus, or soft-kNN label smoothing? This is the real threat. If neighbor voting already captures what locality buys, the Gaussian process machinery adds nothing. The first ablation settles it directly. If noise-aware shrinkage does not beat plain smoothing on continual error, on worst-order accuracy, and on error ranking by uncertainty, we will report that, and the contribution reduces to "residual-on-source kNN." That would still be a meaningful and cheaper result, but a smaller one.

Why not just recompute normalization statistics, or run Tent? The evidence in the file shows both fail in exactly the regime we target. Under CoTTA's continual protocol, Tent-continual (20.7%) is worse than plain normalization updates (20.4%), which is the signature of error accumulation from global updates. MemBN shows normalization updates become erratic at batch sizes of one or two. Our method is per-sample and does not depend on batch statistics. We will include batch-size-1 streams to test this.

Why not just fit class Gaussians in closed form, like ADAPT or OGA? These methods are fast and strong, but they assume one unimodal Gaussian per class, with shared covariance in ADAPT's case. They also model class distributions rather than correcting a trained classifier. They therefore cannot express "trust the source here, override it there." We will compare on the CLIP natural-shift benchmarks, where ADAPT's 66.53% is the bar. We expect our method's advantage to appear on multimodal classes such as ImageNet-R renditions, and we would not be surprised to tie on the others.

Why not just run CoTTA? CoTTA is more accurate when compute is unlimited. It updates every weight, keeps a teacher network, and averages 32 augmentations per step. Our question is whether a frozen, backpropagation-free local correction can recover most of that robustness to forgetting through its structure rather than through stochastic weight resets. The cost–accuracy comparison answers that.

Proposed Experiments and Decisive Checks

The benchmarks are CIFAR-10-C and CIFAR-100-C in both single-shift and CoTTA-style continual form, ImageNet-C at severity 5, and the domain-generalization suite of PACS, VLCS, OfficeHome, and TerraIncognita using T3A's ResNet-50 and TAST's ResNet-18 protocols. We also run ImageNet-A/R/Sketch with a CLIP backbone.

We report error, expected calibration error, area under the risk–coverage curve when selecting by $s^2$, wall-clock time, and memory. Following OGA, we also report expected tail accuracy over the worst 10% of 50 random class orderings. Tail accuracy is the metric that most directly measures the confinement property we claim.

Three results would be decisive. First, on the continual CIFAR-10-C protocol repeated over ten cycles, the residual Gaussian process's error on the first domain should stay flat in later cycles while Tent's degrades. Second, the plain-smoothing ablation should lose at least a point of tail accuracy relative to the full method. If it does not, the Gaussian process is not earning its place. Third, a controlled check on the theory: on CIFAR-10-C we have true labels offline, so we can measure how smooth the true residual $f_t - f_s$ is in feature space for each corruption type. SATL and the model-shift bounds predict our gains should track that smoothness. Finding gains on corruptions where the residual is rough would mean the method works for a reason other than the one we claim.