Image retrieval systems embed every picture as a vector and return the nearest vectors to a query. Small, invisible perturbations to a query or to a database image can reorder those neighbors arbitrarily. The defenses that work today have two costs. They retrain the embedding network against multi-step attacks built from triplets (anchor, positive, negative) inside each mini-batch, which makes training several times slower. They also lose much of their clean retrieval quality: the best robust method on the CUB birds benchmark keeps a clean Recall@1 of only 37.2% (Enhancing Adversarial Robustness for Deep Metric Learning), while non-robust global-objective methods reach 71.7% there, under their own reported setup (Anti-Collapse Loss for Deep Metric Learning).
We propose that both costs come from the same choice: robust metric learning is still built on local, batch-dependent triplet objectives. Our approach has three parts. First, we switch to a global objective, in which each class is represented by one learned prototype vector (a "proxy"). The attack then becomes a single cheap per-sample gradient step. Second, we diagnose the clean-accuracy loss as adversarially accelerated collapse of each class to a point. Third, we counter that collapse directly with a volume-preserving coding-rate term on clean embeddings. The resulting method attacks once per update, trains one network, and should move the robustness–accuracy frontier outward at a fraction of current compute.
Core Claim
In zero-shot retrieval the test classes are unseen during training. Intra-class variation in the embedding therefore carries the information that transfers to new classes, and a representation that collapses each training class to a point discards it. On the Robustness of Neural Collapse and the Neural Collapse of Robustness studies neural collapse, the convergence of last-layer class features to maximally separated, zero-variance points. It shows that standard adversarial training enforces exactly this collapse. It also finds that TRADES reaches comparable robustness while avoiding it, so collapse is sufficient for robustness but not necessary.
Anti-Collapse Loss for Deep Metric Learning separately notes that proxy losses already push intra-class features toward isotropic blobs around the proxy. Putting these together, we hypothesize that naively adversarially training a proxy loss compounds two collapse pressures, and that this compounding explains a large share of clean Recall@1 loss in robust metric learning. If this holds, robustness should be obtainable by constraining adversarial embeddings to stay near their clean counterparts, as TRADES does, rather than near proxies. Clean accuracy should then be protected by explicitly keeping embedding volume large.
Method
Let $z_\theta(x)$ be the normalized embedding and $\mathcal{L}_{PA}$ the Proxy Anchor loss. Proxy Anchor treats each class proxy as an anchor and pulls or pushes all batch samples relative to it, at cost linear in batch size times active classes (Proxy Anchor Loss for Deep Metric Learning). The perturbation $\delta$ is one signed-gradient step on $\mathcal{L}_{PA}$ from a uniform random start inside the $\epsilon$-ball. This is the recipe of Fast is better than free, reused unchanged. Because $\mathcal{L}_{PA}$ depends on the sample and the fixed proxies rather than on which other samples share the batch, the attack target is stable across batches. Triplet attacks lack this property. Training minimizes
$$\mathcal{L} = \mathcal{L}_{PA}\big(z_\theta(X)\big) + \lambda \,\big\| z_\theta(X+\delta) - z_\theta(X) \big\|^2 - \gamma \, R\big(z_\theta(X)\big), \quad R(Z) = \tfrac{1}{2}\log\det\!\big(I + \tfrac{d}{n\eta^2} Z^\top Z\big).$$
Here $X$ is the batch of $n$ images and $Z$ the $n\times d$ matrix of their clean embeddings. $\lambda$ weights embedding consistency, which is the TRADES idea moved from predictions to embeddings (Theoretically Principled Trade-off between Robustness and Accuracy). $R$ is the coding rate, a measure of the volume spanned by the batch. $\gamma$ weights it, and $\eta$ is the precision constant of the coding-rate principle. The $d\times d$ determinant at $d=512$ is negligible next to a backbone pass. Adding a proxy-level coding-rate term, as in the anti-collapse paper, is a cheap variant we also test.
The adversarial term never pulls perturbed samples toward a proxy. It only asks them to stay where their clean version is. Neither the proxy pull nor the adversarial term is allowed to shrink intra-class spread unopposed, because the coding-rate term pushes back on volume loss. Training cost is two forward–backward passes per step, against roughly $K+1$ for $K$-step PGD triplet methods.
Research Program
The work has three parts.
Diagnostic part. We train Proxy Anchor with standard PGD adversarial training on CUB-200-2011, Cars-196, and Stanford Online Products. We measure intra-class variability relative to between-class spread, on both training classes and held-out test classes, across training. We test whether this collapse measure predicts the clean Recall@1 gap better than robustness level does.
Method part. We build the objective above and map its frontier over $\lambda$ and $\gamma$.
Stability part. Single-step training is known to fail abruptly through catastrophic overfitting. Understanding Catastrophic Overfitting in Single-step Adversarial Training traces this failure to distorted decision boundaries, signalled by input-gradient norm spikes of up to 35×. We do not know whether an embedding-space analogue occurs. We will log input-gradient norms and multi-step retrieval robustness every epoch. If collapse occurs, we adopt that paper's adaptive step magnitude as a remedy and do not claim the coding-rate term prevents it.
Baselines
The naive reference points are two. The first is a clean Proxy Anchor model with no defense. The second is Proxy Anchor trained with plain random-start single-step attacks and no consistency or coding-rate terms. These show what fast adversarial training on a global loss gives for free.
The strongest existing method is Hardness Manipulation with gradual adversary and intra-class structure loss, from Enhancing Adversarial Robustness for Deep Metric Learning. It perturbs triplets to a target hardness with PGD. On CUB it reports an Empirical Robustness Score of 33.5 with clean Recall@1 of 37.2%. That score aggregates the ranking-attack suite of Adversarial Attack and Defense in Deep Ranking. The paper whose Anti-Collapse Triplet defense the file also lists reaches an Empirical Robustness Score of 31.3 and Recall@1 of 30.6%. Both have to be beaten on their own metric, which is why they are the right comparison. We will rerun both from released settings under one backbone, embedding size, augmentation, epoch count, and $\epsilon$. Our method gets the same hyperparameter-search budget in trials as each baseline, and we report wall-clock GPU hours for every run.
Beating Hardness Manipulation means one of two outcomes on CUB and Cars. Either a higher robustness score at equal or better clean Recall@1, or higher clean Recall@1 at equal robustness score, in both cases with lower training time. A clean Recall@1 gain of several points at matched robustness would be the headline result. A one-point gain would not.
Two ablations isolate the key ingredients. The first removes the coding-rate term ($\gamma=0$). The second replaces the embedding-consistency term with a standard adversarial proxy loss, so that perturbed samples are pulled to proxies.
Why not just…?
Why not just run fast single-step adversarial training on Proxy Anchor and stop? This is the real threat. The global loss alone might already deliver most of the speedup and robustness. Our first naive baseline is exactly this, so the experiment settles the question directly. If it matches our method on both clean Recall@1 and robustness, the collapse hypothesis is wrong for this setting and the contribution reduces to an empirical finding about efficient robust retrieval. We would report it as such.
Why not just use fewer PGD steps or single-step attacks inside the triplet-based Hardness Manipulation method? Triplet attack objectives change with batch composition, and triplet mining scales cubically in batch size. Neither property is fixed by taking fewer steps. We will nonetheless run a single-step Hardness Manipulation variant at matched compute, since the original paper names fast training as future work.
Why not preserve clean structure with a clean teacher network, as in topology-preserving adversarial training? Topology-preserving Adversarial Training aligns neighbor graphs between a clean and an adversarial model and gains up to 8.78% natural accuracy on CIFAR-100 combined with TRADES. It requires two full networks and still runs multi-step attacks. We will implement its neighborhood-alignment loss in place of our coding-rate term at matched compute. Whichever term wins on the Recall@1–robustness frontier per GPU-hour wins.
Why not skip adversaries entirely with a margin-widening loss like Guided Complement Entropy? Improving Adversarial Robustness via Guided Complement Entropy costs nothing extra but holds only about 15% accuracy under iterative attack in classification. The file also reports that it fails under AutoAttack. We include an adversary-free variant of our objective, with $\lambda=0$, to measure this gap in retrieval.
Decisive Checks
We evaluate on CUB-200-2011, Cars-196, and Stanford Online Products. Metrics are clean Recall@K and NMI, the candidate and query ranking attacks, the Empirical Robustness Score, and training GPU-hours. The idea succeeds only if four conditions hold.
First, the collapse measure on test classes must track clean Recall@1 loss across methods, and adding the coding-rate term must raise both.
Second, the full method must dominate Hardness Manipulation's frontier at lower compute.
Third, robustness must survive attacks designed not to exploit our training signal, so that gradient masking is ruled out. Reliable evaluation of adversarial robustness warns that fast defenses often fail under parameter-free and query-based attacks. We therefore attack the ranking directly with many-step step-size-free PGD rather than through proxies. We also attack with transfer from a PGD-trained surrogate and with a query-only score-based attack.
Fourth, training must show no late collapse of multi-step robustness.
A sanity transfer to CIFAR-10 under AutoAttack, using the embedding as a nearest-proxy classifier, would show whether the effect extends beyond retrieval.
References
- Enhancing Adversarial Robustness for Deep Metric Learning
- Anti-Collapse Loss for Deep Metric Learning
- On the Robustness of Neural Collapse and the Neural Collapse of Robustness
- Proxy Anchor Loss for Deep Metric Learning
- Fast is better than free: Revisiting adversarial training
- Theoretically Principled Trade-off between Robustness and Accuracy
- Understanding Catastrophic Overfitting in Single-step Adversarial Training
- Adversarial Attack and Defense in Deep Ranking
- Topology-preserving Adversarial Training for Alleviating Natural Accuracy Degradation
- Improving Adversarial Robustness via Guided Complement Entropy
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks