Proxy Profiles: Test-Time Adaptation for Image Retrieval on Unseen Classes

Image retrieval systems are trained to place pictures of the same kind of object close together in an embedding space, a vector space where distance stands for similarity. They are then asked to search over classes they never saw in training, often on photos taken in different conditions: blur, fog, sensor noise, new cameras. Methods that adapt a model during deployment, using only the unlabeled test images, have made classifiers much more robust to these shifts. None of them carries over to retrieval, because they all work by sharpening the model's predictions over its known training classes. In retrieval the test classes are new, so there is nothing to sharpen. Worse, forcing confident predictions onto unfamiliar inputs pushes them into the wrong clusters.

We propose a fix that reuses something retrieval models already have. Modern scalable retrieval losses replace the cubic number of sample triplets with a small set of learned class representatives called proxies, one vector per training class. We argue that these frozen proxies form a compact reference frame for unseen classes. Any image, whatever its class, can be described by its soft similarity to each of the training proxies, and we call this its proxy profile. Adapting the network at test time so that proxy profiles stay stable across views and match their training-time statistics gives a label-free signal that never asks "which known class is this?" We direct this adaptation at the one component most likely to spread corruption: the global-context block that lets every pixel draw on every other pixel.

Core Claim

The claim has three parts. First, proxies learned by a scalable metric loss can serve as a source-free anchor for test-time adaptation in open-set retrieval. They cost $O(C)$ memory for $C$ training classes and need no stored source images. Second, the right parameters to adapt are the normalization affine parameters together with a gate on the global-context pathway. The reason is that long-range routing, which fixes the narrow view of convolutions, is also the route by which a localized corruption reaches clean regions. Third, both parts are needed. If simply re-estimating normalization statistics recovers the same retrieval accuracy, the proxy-profile objective is unnecessary and we will say so.

The proposal builds on four existing lines of work. Proxy Anchor Loss supplies the training objective. It treats each proxy as an anchor that pulls in all positives in a batch and pushes away all negatives, which avoids triplet mining. Tent supplies the choice of adapting only normalization scale and shift parameters, but its entropy objective is exactly what we replace. UniEnt motivates the replacement. It shows that entropy minimization on unknown-class inputs falls into a "confidence trap," with Tent's unknown-detection AUROC at 65.00% on CIFAR-100-C, and it fixes this for classification by separating known from unknown samples. In retrieval every test sample is unknown, so a separation step cannot help. Leveraging Proxy of Training Data for Test-Time Adaptation is the closest relative. It preserves an inter-class similarity matrix taken from classifier weights during adaptation, but it uses condensed synthetic images and tests closed-set classification only. Our proxies live in the embedding space itself, and we use them to describe unseen classes rather than to label known ones.

Research Program

The backbone is a ResNet-50 with one Recurrent Criss-Cross Attention block, taken from CCNet, inserted before global pooling. In this block each pixel attends along its row and column, twice, so it reaches the whole image at $O(N\sqrt{N})$ cost for $N$ pixels. The block's residual output is multiplied by a learnable per-channel gate $g$, initialized to 1. Training uses Proxy Anchor loss on the standard seen-class splits. We keep two read-outs: a local embedding $f_L(x)$, pooled before the attention block, and a global embedding $f_G(x)$, pooled after it.

With frozen, normalized proxies $P \in \mathbb{R}^{C \times d}$, where $d$ is the embedding dimension, the proxy profile of image $x$ under read-out $f$ is

$$p_f(x) = \mathrm{softmax}\big(P\, f(x)/\tau\big),$$

where $\tau$ is a temperature. At deployment the model receives an unlabeled gallery or query batch $B$ from a shifted distribution. It updates only the normalization affine parameters and the gate $g$, by minimizing

$$\mathcal{L} = \frac{1}{|B|}\sum_{x\in B} \mathrm{KL}\big(p_{f_G}(x)\,\|\,p_{f_L}(\tilde{x})\big) + \lambda\, \mathrm{KL}\big(\bar{p}_{\text{src}}\,\|\,\bar{p}_B\big).$$

Here $\tilde{x}$ is a mild augmentation of $x$, $\bar{p}_B$ is the batch-average profile, $\bar{p}_{\text{src}}$ is the average profile recorded on held-out training images, which is a single $C$-dimensional vector, and $\lambda$ weights the second term.

Each term has a job. The first asks the globally routed view to agree with a local, augmented view. If corruption is being spread through the attention block, it shows up as disagreement, and the gate can turn the routing down. The second term stops the trivial solution in which every image collapses onto one proxy, because it ties the overall spread of profiles to training-time statistics. It plays the role that marginal-entropy terms play in UniEnt, but it does not need class predictions. The proxies stay frozen throughout, which avoids the batch-to-batch proxy drift that Variational Continual Proxy-Anchor found during training. That paper's fix required Newton updates that slowed training by 38%.

The analysis has three parts. Before any adaptation, we measure whether the attention block helps or hurts retrieval under each corruption type and whether the effect depends on how localized the corruption is. Gap 1 in the file says this is unknown. We then run the adaptation and record the learned gate values per corruption, which tests directly whether the method turns routing down where the corruption is spatially local. Finally, we test single-image query streams, the regime where MemBN reports that batch statistics break down.

Baselines

The naive reference is the trained Proxy Anchor model with the attention block, evaluated unchanged on corrupted unseen-class data. Its gap from clean accuracy defines the room for improvement.

The first adaptive baseline is normalization statistics re-estimation, following The Norm Must Go On. It recomputes batch-norm means and variances on the test batch, with no gradients. For single-image streams we use the MemBN memory variant. This is the most dangerous competitor because it is free, it needs no class predictions, and so it applies to retrieval as is. We run it on the same model and the same batches. Beating it means a Recall@1 gain on the corrupted unseen classes that holds up across seeds, averaged over corruption types and severities.

The second baseline is Tent adapted to retrieval: entropy of the proxy-softmax, minimized over the same parameters with the same learning-rate search budget. We expect it to degrade through the confidence trap. The third is a T3A-style method, after T3A, which builds centroids from confident test samples without backpropagation. Here the centroids come from clustering the test gallery, since there are no known-class predictions to filter on.

The strongest existing training-time method is PA+DADA. It aligns proxies and features adversarially during training and reports clean Recall@1 of 72.9% on CUB-200-2011 and 92.1% on CARS196 with a ResNet-50. We retrain it with our backbone and evaluate it under corruption, both unadapted and with our test-time step added on top. No reported result was found on Lacuna for any metric-learning method on corrupted CUB or CARS images.

Three ablations isolate the key ingredients. The first removes the attention block entirely, giving a local-only backbone. The second keeps the block but freezes the gate during adaptation. The third drops the local-global agreement term and keeps only the marginal-matching term. The method must beat every baseline under the same search budget, and the ablations must show that each ingredient contributes.

Why not just…?

Why not just re-estimate batch-norm statistics? This is the real threat. Tent's own ImageNet-C numbers show statistics replacement at 49.9% error against Tent's 44.0%, so optimizing affine parameters added something on top of statistics alone, but that was for classification. For retrieval the comparison has not been run. If re-estimation matches our method within noise, we will conclude the proxy-profile objective adds nothing. We will report it as the recommended baseline and keep only the receptive-field analysis.

Why not just use Tent with the proxies as a classifier? UniEnt's 65.00% AUROC for Tent on CIFAR-100-C shows that entropy minimization mistreats unknown inputs. In retrieval all test classes are unknown, so we expect retrieval geometry to collapse rather than improve. Including Tent as a baseline measures this directly rather than assuming it.

Why not just drop global attention, or train with heavy augmentation? If the attention block spreads corruption, the simplest fix is a local backbone. CGNL reports that global routing helps on clean fine-grained data, raising CUB classification from 84.05% to 85.68%, so dropping it has a cost. The local-only ablation will show whether that cost outweighs the robustness gain. Augmentation-heavy training is compatible with our method, and we apply it to every model equally.

Why not just cluster the test gallery, T3A-style? Clustering gives centroids, but it does not repair features that the shift has already distorted. That is why we include it as a baseline rather than treating it as an alternative design.

Proposed Experiments

Training uses the standard seen-class halves of CUB-200-2011 and CARS196. Testing applies the 15 ImageNet-C corruption types at 5 severities to the unseen-class halves, and we will release these corrupted sets. Stanford Online Products serves as a test of how the method scales to large class counts, since the proxy reference frame grows with $C$. The metrics are Recall@1 and Recall@4, adaptation latency, and memory, plus clean-data Recall@1 after adaptation to check that the method does not forget what it learned in training.

Three results would decide the question. The first is whether, without adaptation, the attention-block model loses more Recall@1 than the local-only model on spatially localized corruptions than on global ones. The second is whether our adaptation beats statistics re-estimation, with the gate moving toward lower values exactly where the first result shows harmful routing. The third is whether the gains survive single-image streams. If the first fails, the receptive-field half of the story collapses to a plain test-time adaptation method. If the second fails, the proxy-profile objective is unnecessary. Either outcome is informative, and we will report it as found.