Image retrieval systems learn an embedding space in which pictures of the same thing land close together and pictures of different things land far apart. Two separate fixes have each helped this kind of training. The first replaces costly searches for "hard negatives" (confusable examples from other classes) with learnable class centers called proxies. The second adds an auxiliary task, such as predicting how much an image was rotated, which regularizes the embedding. These fixes pull against each other. Retrieval wants the embedding to ignore rotation, while the auxiliary task wants it to encode rotation. Every existing combination settles the conflict by hand-tuning a weight between the two losses.
We propose to remove the conflict by design instead of balancing it. The embedding is split into a part that cannot change under rotation and a part that rotates in a known, fixed way. Each class proxy becomes an orbit: the class center together with its rotated copies. One proxy-style loss then trains class identity and orientation at the same time, with no pair mining, no separate rotation head, and no loss weight to tune. The central testable prediction is diagnostic. Gradient de-confliction methods from multi-task learning should help the usual "proxy loss plus rotation head" design noticeably, and should help the orbit-proxy design almost not at all. That would show the conflict was caused by an entangled representation, not by a weak optimizer.
Core Claim
The tension between invariant metric learning and rotation-sensitive auxiliary tasks is a representation problem.
Prior work treats it as an optimization problem, or ignores it. Unsupervised Deep Metric Learning via Auxiliary Rotation Loss (UDML-SS) pairs a pair-based metric loss with a 4-way rotation classifier on a shared Inception backbone and fixes their weight by hand. Its ablation shows the rotation loss matters: removing it costs about 7% Recall@1 on Cars-196. Equivariant Contrastive Learning does the same thing on top of a contrastive loss (InfoNCE) that compares every augmented view against every other. Both keep the pairwise cost and the hand-set weight.
On the proxy side, Proxy Anchor Loss for Deep Metric Learning removes mining by comparing each sample only to class proxies. It reaches better Cars-196 Recall@1 in about 20 minutes than pair-based methods reach in 100 minutes. Non-isotropy Regularization for Proxy-based Deep Metric Learning (NIR) shows these proxies collapse each class into a structureless ball around its center, and adds a normalizing-flow term, again with a fixed weight. The resulting structure is learned freely, not tied to any physical transformation.
We claim that if the embedding carries orientation in a dedicated subspace with a known rotation rule, and proxies are orbits in that subspace, three things should follow. Rotation supervision should restore intra-class structure, playing the role NIR plays. Training cost should stay at proxy level. Gradient conflict between identity and orientation should largely disappear without any balancing algorithm.
Method
An encoder maps image $x$ to $z = (u, v)$. The part $u \in \mathbb{R}^{d_u}$ is meant to be unchanged by 90° rotations. The part $v \in \mathbb{R}^{d_v}$ is meant to transform as $v(g\cdot x) = \rho(g)\, v(x)$, where $g$ is one of the four 90° rotations and $\rho(g)$ is a fixed block matrix that rotates pairs of coordinates by multiples of 90° and flips sign channels.
Each class $c$ has a proxy $(a_c, b_c)$. Its orbit is the set $\{(a_c, \rho(h) b_c)\}$ over the four rotations $h$. For an input rotated by $g$, the score against orbit element $(c, h)$ is
$$s(x, c, h) = \cos(u, a_c) + \beta \cos\big(v, \rho(h) b_c\big),$$
where $\beta$ sets how strongly orientation enters the score. This $\beta$ enters the score, not a sum of two losses.
We apply the Proxy Anchor log-sum-exp loss over the $4C$ orbit elements. The single positive is $(c, g)$. The other three rotations of the correct class serve as hard negatives in the orientation subspace. They are known analytically, so finding them requires no search. Cost stays $O(N \cdot 4C)$ for $N$ samples, against the $O(N^2)$ or worse of pair mining.
Retrieval uses only rotation-invariant quantities: $u$ concatenated with the per-block norms of $v$.
We build the encoder in two variants. The architectural variant uses a 4-fold rotation-equivariant backbone assembled from the slice, pool, roll and stack operations of Exploiting Cyclic Symmetry in Convolutional Neural Networks, in the spirit of ReDet's ReResNet from ReDet: A Rotation-Equivariant Detector for Aerial Object Detection. Here $u$ comes from pooling over orientation channels, so $u$ is exactly invariant and $v$ exactly equivariant by construction. The learned variant uses a standard backbone. The rule $v(g \cdot x) = \rho(g)v(x)$ is then encouraged only through the orbit loss itself, much as Steerable Equivariant Representation Learning learns latent operators that mirror input augmentations. That work learns a separate map per augmentation and never connects to proxies. We fix $\rho$ in advance and make it part of the proxy geometry.
Research Program
The program has three parts.
Build. We implement orbit proxies in both encoder variants inside a standard Proxy Anchor training codebase, keeping that paper's backbone, embedding size, batch size and schedule wherever possible.
Analyze. We measure, throughout training, the cosine between the backbone gradients of the identity term and the orientation term. We compute this once with $\beta$ split out as a separate loss and once for the entangled baseline. We expect the factored design to show cosines near zero or positive where the entangled design shows sustained negative values.
We also test the timing finding of Metric Learning for Clifford Group Equivariant Neural Networks, which had to switch on its learned metric at about 90% of training to avoid destabilizing optimization. If the orbit loss is stable from step zero, that supports the claim that a fixed, known transformation rule avoids the instability a freely learned one causes.
Test regimes. We test in two regimes that differ in how much rotation invariance the data actually requires. The first is the standard fine-grained benchmarks CUB-200-2011, Cars-196 and Stanford Online Products. Their images are mostly upright, so rotation acts as a regularizer. The second is an aerial-object retrieval set we cut from DOTA instance crops (DOTA: A Large-Scale Dataset for Object Detection in Aerial Images), with disjoint train and test categories. There, objects appear at arbitrary orientations and invariance is a genuine requirement.
Baselines
Proxy Anchor (naive reference). The do-nothing reference is plain Proxy Anchor with standard augmentation. As reported inside the NIR paper, it reaches 82.4% Recall@1 on Cars-196 and 64.4% on CUB-200-2011. We rerun it in our codebase so every method shares data splits, backbone, image resolution, epochs and a matched hyperparameter search budget, measured in number of trials. Our reproduction must land near those figures before any comparison counts.
Proxy Anchor plus NIR (strongest supervised competitor). The strongest existing supervised method in the file is Proxy Anchor with Non-isotropy Regularization, at 85.2% Recall@1 on Cars-196 and 66.0% on CUB-200-2011 (Non-isotropy Regularization for Proxy-based Deep Metric Learning). It is the right competitor because it targets the same failure we target, isotropic collapse around proxies, but with an unstructured flow. We run it with its fixed weight searched under the same trial budget.
Beating it means two things. Orbit proxies must exceed its upright Recall@1 on Cars-196 and CUB averaged over at least three seeds. They must also clearly beat it on rotated-query retrieval, with no added flow network.
For Stanford Online Products, the file has no reported Proxy Anchor or NIR number (no reported result found on Lacuna for SOP Recall@1 of either method), so we rely on our own reruns. For rotated-query retrieval and for DOTA-crop retrieval there is no reported result found on Lacuna for any method; all numbers there are ours.
Entangled rotation-head design, with and without CAGrad (the obvious combination). This baseline is Proxy Anchor plus a separate 4-way rotation classifier head, combined with a grid-searched weight. We run it once with ordinary gradient descent and once with Conflict-Averse Gradient Descent for Multi-task learning (CAGrad). CAGrad picks the update that most improves the worse-off task while staying near the average gradient. It cut NYU-v2 average per-task degradation to 0.20% against 3.97% for PCGrad, but has never been applied to a metric loss paired with a geometric task.
Unsupervised regime. We swap proxies for k-means pseudo-label proxies and compare against UDML-SS. It reports 54.7%, 45.1% and 63.5% Recall@1 on CUB, Cars and SOP (Unsupervised Deep Metric Learning via Auxiliary Rotation Loss).
Ablations isolating the key ingredient. Setting $\beta = 0$ removes orientation entirely. The non-factored orbit variant rotates the whole $z$ under $\rho$, so no invariant part exists. Finally, we cross each encoder variant with and without CAGrad.
Why not just…?
Why not just tune a fixed weight on a rotation head, or add CAGrad on top? This is the obvious combination, and it is included as a baseline rather than argued away. The evidence against fixed weights is indirect. Task distraction and task neglect are documented across domains, for example in End-to-End Thorough Body Perception for Person Search. CAGrad's NYU-v2 gains show that de-confliction can matter. Neither result is specific to metric learning.
Our gradient-cosine measurements and the two-by-two experiment (factored or entangled, crossed with CAGrad or plain descent) settle the question. If entangled-plus-CAGrad matches orbit proxies on accuracy, the representational argument fails and the cheaper story wins. CAGrad's own cost also scales with the number of tasks, while orbit proxies add essentially nothing.
Why not just train Proxy Anchor with random rotation augmentation? This is a real threat on the rotated-query test, since augmentation directly targets it. Learning rotation invariant convolutional filters for texture classification argues that augmentation blurs filters, because a single filter must cover several orientations. ZZ-Net: A Universal Rotation Equivariant Architecture for 2D Point Clouds reports degradation on orientations under-represented in training. Augmentation also pushes toward full invariance, which is exactly the isotropic collapse NIR identified. We therefore expect augmentation to buy rotated-query robustness at a cost in upright Recall@1. Running the augmentation baseline on both the upright and rotated protocols tests this directly.
Why not just use a rotation-equivariant backbone with orientation pooling and plain Proxy Anchor? This is the most serious alternative. It gives exact invariance with no auxiliary task, and ReDet showed such backbones are compact: 121 Mb against 313 Mb, at 80.10 mAP on DOTA-v1.0.
The orbit design differs only by keeping $v$ and the orientation term, so the $\beta = 0$ ablation on the equivariant encoder is exactly this baseline. If it matches the full method, we will report that architecture alone resolves the conflict and that orbit proxies add nothing. That would be a clean negative result about the value of orientation supervision once invariance is built in.
Why not just pre-train with rotation prediction and then fine-tune with proxies? Two-stage training avoids balancing altogether. However, UDML-SS found rotation helps as an ongoing regularizer during metric training, and the Clifford result shows that where to place a stage boundary is a hand-chosen hyperparameter. We include this pipeline with the same total epoch budget.
Proposed Experiments
The primary metric is Recall@1, with Recall@k reported alongside, on CUB-200-2011, Cars-196 and SOP under the standard disjoint-class splits. We evaluate under two protocols: upright queries and galleries, and queries rotated by a random multiple of 90° against an upright gallery. We report orientation-recovery accuracy by reading the nearest orbit element. Wall-clock time to reach the baseline's best Recall@1 is measured on identical hardware, including a pair-based Multi-Similarity loss to anchor the mining-cost comparison against Proxy Anchor's reported 20-versus-100-minute gap. The DOTA-crop retrieval set tests the regime where invariance is truly required.
Three checks decide the claim.
The first is the interaction test. CAGrad should improve the entangled baseline by a margin larger than seed variance, and should change the orbit design by less than seed variance. Measured gradient cosines should line up the same way.
The second is the $\beta = 0$ ablation on the equivariant encoder. Orbit proxies must beat pure architectural invariance on upright Recall@1 for the orientation term to justify its place.
The third is the stability test. Orbit training should converge from step zero without the late switch-on the Clifford-network work needed. If the first check fails, the paper becomes a study showing that de-confliction matters for metric learning. If the second fails, it becomes a study showing that equivariant backbones make auxiliary rotation tasks redundant. Either outcome is informative, and the experiments are set up so that either can be reported cleanly.
References
- Unsupervised Deep Metric Learning via Auxiliary Rotation Loss
- Equivariant Self-Supervised Learning: Encouraging Equivariance in Representations
- Proxy Anchor Loss for Deep Metric Learning
- Non-isotropy Regularization for Proxy-based Deep Metric Learning
- Exploiting Cyclic Symmetry in Convolutional Neural Networks
- ReDet: A Rotation-Equivariant Detector for Aerial Object Detection
- Steerable Equivariant Representation Learning
- Metric Learning for Clifford Group Equivariant Neural Networks
- DOTA: A Large-Scale Dataset for Object Detection in Aerial Images
- Conflict-Averse Gradient Descent for Multi-task learning
- End-to-End Thorough Body Perception for Person Search
- Learning rotation invariant convolutional filters for texture classification
- ZZ-Net: A Universal Rotation Equivariant Architecture for 2D Point Clouds