Learning to Retrieve the Rival: End-to-End Contrastive Neighbor Selection for Fine-Grained Recognition

In fine-grained recognition, telling a California Gull from a Ring-billed Gull or one Boeing 737 variant from another, the hard part is not separating a query from all other classes. It is separating it from the two or three sibling classes it closely resembles. The most successful exemplar method in this setting works in two separate steps. It takes the classifier's few most likely classes, pulls the nearest training image from each, and uses a separately trained comparison network to decide which exemplar truly matches. This raises accuracy and helps human users, but the classifier is frozen and the lookup is a discrete search, so the comparison network's knowledge never improves the features used to find the neighbors. We propose to make that whole loop trainable. For each of the model's top candidate classes, a softly weighted nearest exemplar is selected within that class. A lightweight part-level comparison is run against each selected exemplar, and the loss is backpropagated through the comparison, through the soft neighbor weights, and into the shared backbone.

In two sentences: the model's prediction is computed from real training images of the classes it currently confuses, and training pushes the representation to make those specific rival retrievals informative. This differs from classic differentiable neighbor methods, which spread a softmax over every point in the dataset. It also differs from prototype methods, which reason over synthetic vectors rather than actual images.

Core Claim

Differentiable neighbor selection works best under low inter-class variance when it is class-conditional and restricted to the confusable classes, not global. Let $q = f_\theta(x)$ be the backbone's global embedding of query $x$ and $T(x)$ the set of the $K$ classes with highest classifier logit $a_c(x)$. Let $N_c(x)$ be the $m$ training exemplars of class $c$ closest to $q$, found in a memory bank of embeddings. The class score is then

$$s_c(x) = a_c(x) + \beta \sum_{j \in N_c(x)} w_{cj}\, h_\phi(x, x_j), \qquad w_{cj} = \frac{\exp(-\|q - z_j\|^2/\tau)}{\sum_{l \in N_c(x)} \exp(-\|q - z_l\|^2/\tau)},$$

where $z_j = f_\theta(x_j)$ is the neighbor's embedding recomputed with the current backbone, $h_\phi$ is a small cross-attention module that compares the query's patch features to the neighbor's patch features, $\tau$ is a temperature, and $\beta$ weights the comparison term. Training minimizes cross-entropy of $s_c$ over $T(x)$ together with ordinary cross-entropy on $a_c$.

Three properties follow from this design. The within-class softmax runs over only $m$ exemplars, so its support is bounded and it avoids the dilution in which every visually similar image in a large gallery gets a little probability mass. Because the rivals are the model's own top-$K$ confusions, gradient signal concentrates on sibling boundaries rather than easy classes. And the displayed exemplars are the ones the prediction is actually computed from.

The claim is falsifiable in three ways. It fails if a frozen-backbone version with the same comparison module matches it, if global softmax over all exemplars matches it, or if picking rival classes at random matches picking the top-$K$.

Relation to Prior Work

Neighbourhood Components Analysis supplies the core mechanism. It learns a transform by maximizing the expected leave-one-out accuracy of a softmax neighbor rule over all points. We reuse the softmax weighting but restrict it within class and to confusable classes. Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees extended soft neighbor selection to deep features through stochastic tree traversal. It reached 1.85% error on MNIST with 202 nodes, but it was never tested where classes differ subtly, and its branching search is hard to run efficiently on a GPU. Our selection is a batched masked softmax.

This Looks Like It Rather Than That: ProtoKNN is the closest end-to-end competitor. It classifies by nearest neighbors over learned prototypes, with Sinkhorn-Knopp balancing and Gumbel-Max prototype assignment, and reports 87.5% on CUB-200-2011 with ResNet-50. Its decisions rest on free latent vectors under one global metric. Ours rest on real training images, compared part-to-part by a learned module. This Looks Like That motivates case-based reasoning, but it ties prototypes to classes through a fixed linear layer and needs an offline projection stage. We have neither.

PCNN motivates the proposal most directly. Retrieving one neighbor per probable class and reranking with a CrossViT comparator raised ResNet-18 on CUB from 60.22% to 71.09%, and raised human accuracy from 54.55% to 64.58%. The same paper also constrains our claims. Its gains shrink to roughly 1–3% on Vision Transformers, and inference was 7.3x slower (about 2.9x with class pruning). We keep its "probable-class" insight but remove the frozen classifier, the discrete lookup, and the second full network.

Metric-learning work on fine-grained data motivates dropping the single global metric. Multi-stage metric learning learned a linear metric on frozen features and reached 65.84% on CUB. Hierarchical Metric Learning raised NWPU-RESISC45 from 72% to 84.6% with node-specific metrics, but its features are frozen and its routing is hard and irreversible. Our part-level comparator acts as a query-dependent local metric that is trained jointly with the features.

The attention-based methods set the accuracy bar and supply the evidence that discriminative cues are local. MGE-CNN reports 89.4% on CUB. AP-CNN reports 88.4% on CUB, 95.4% on Cars, and 94.1% on Aircraft. Cross-X reports 86.4% on NABirds. DPS-CNN found that the throat region alone gives 68.7% accuracy while the tail gives 20.0%. None of these methods explain a decision by pointing to a training image.

Research Program

We will build one shared backbone. For the main runs this is a ResNet-50 so that comparison with ProtoKNN and AP-CNN is direct, with a ViT-B replication. A momentum-updated memory bank will store global embeddings of all training images. It is used only to find $N_c(x)$ for each candidate class.

The selected $K \cdot m$ neighbors are re-encoded with the current backbone, with gradients enabled, which lets the loss reshape neighbor embeddings as well as the query's. The comparator $h_\phi$ is two cross-attention layers over the shared patch tokens, not a second backbone. At inference, neighbor patch tokens are cached, so the extra cost is $K \cdot m$ small cross-attentions. We start with $K=5$ and $m=3$.

Re-encoding neighbors during training is the main cost. We will control it by subsampling $m$ during training and by warm-starting from a cross-entropy checkpoint. Wall-clock time will be reported against all baselines.

The analysis has three parts. We will track how confusion between the baseline's most-confused class pairs changes during training. We will measure top-$K$ recall of the true class, since the rerank cannot recover a class outside $T(x)$. And we will test whether the retrieved exemplars are faithful to the decision.

Baselines

The naive reference is a backbone of the same architecture trained with cross-entropy and evaluated by its softmax. A second naive reference runs cosine $k$NN on that model's penultimate features, which tests whether any neighbor structure beyond ordinary training helps.

The strongest exemplar baseline is PCNN (PCNN), reproduced on our backbone with its CrossViT comparator trained on hard negative pairs. Its reported number is ResNet-18 on CUB-200-2011, 60.22% to 71.09% top-1. To keep the comparison fair, we give it the same training images, the same total GPU-hours including its comparator, and the same hyperparameter search budget. Beating it means higher top-1 at equal or lower inference latency, where PCNN reports 7.3x (2.9x pruned) over its base classifier.

The strongest end-to-end prototype baseline is ProtoKNN (ProtoKNN), reported at 87.5% on CUB with ResNet-50. It will be rerun with our augmentation and training schedule. Beating it means exceeding its rerun number by more than seed variance, averaged over three seeds.

For raw accuracy, AP-CNN's reported 88.4% CUB, 95.4% Cars, and 94.1% Aircraft, and MGE-CNN's 89.4% on CUB, are the numbers to approach. We do not claim to beat these multi-crop cascades on accuracy alone. We claim to come within about a point of them in a single pass while also producing grounded exemplars.

A deep metric learning baseline trains a supervised contrastive or Proxy Anchor loss with hard-negative mining and classifies with $k$NN. It is tuned with the same budget. No reported number was found on Lacuna for standalone Proxy Anchor on CUB.

Three ablations isolate the key ingredient. The first freezes the backbone and trains only $h_\phi$, which is a shared-backbone PCNN. The second replaces the class-conditional top-$K$ softmax with a global neighborhood-components softmax over the whole bank. The third draws the $K$ rival classes at random instead of from $T(x)$. The proposal is supported only if the full model beats all three.

Why not just…?

Why not just use PCNN on a strong backbone? This is the most serious alternative. PCNN's own gains on ViTs are only 1–3%, so a strong frozen backbone plus reranking may leave little room. The frozen-backbone ablation settles this directly. If it matches the end-to-end model on ViT-B, the accuracy claim is wrong for strong backbones. The remaining contribution would then be the latency reduction from sharing the backbone and dropping the separate CrossViT, and we would report it that way. The end-to-end model's additional possible advantage is higher top-$K$ recall, because training can move true classes into $T(x)$. We measure this separately.

Why not just train a contrastive metric-learning loss and run $k$NN? Contrastive losses already pull same-class points together. But they compare global embeddings under one metric, and the file's evidence suggests global metrics stall on confusable classes: 72% versus 84.6% for localized metrics in Hierarchical Metric Learning. ProtoKNN also needed more than Proxy Anchor loss to reach 87.5%. Still, a well-tuned hard-negative contrastive loss is close in spirit to our rival selection. The global-softmax and random-rival ablations test whether class-conditional selection among the model's own confusions adds anything beyond hard-negative mining.

Why not just use an attention cascade like MGE-CNN or AP-CNN? These methods currently hold the higher reported accuracies, and we may not surpass them. What they cannot do is justify a prediction with a real training image whose parts were compared to the query. PCNN's human study shows that such contrastive exemplars change human accuracy by about ten points. We therefore judge against them on accuracy per unit of inference cost and on faithfulness, not on accuracy alone.

Proposed Experiments

The main benchmarks are CUB-200-2011, Stanford Cars, FGVC-Aircraft, Stanford Dogs, and NABirds, using standard splits and top-1 accuracy. For Stanford Dogs, no reported result was found on Lacuna for the baselines, so all Dogs numbers will come from our reruns.

Beyond overall accuracy, we report three further measures. The first is accuracy restricted to test images whose true class belongs to one of the 20 most-confused class pairs under the cross-entropy baseline. This is where low inter-class variance bites, and where we predict the largest gap over contrastive $k$NN. The second is top-$K$ recall of the true class. The third is inference latency relative to the cross-entropy backbone.

Faithfulness is tested by counterfactual swapping. We replace the highest-weighted rival exemplar with a random image of the same class and measure how often the prediction flips. For our model this should matter. For PCNN's decoupled pipeline it should matter less for the classifier term.

The decisive checks are the three ablations, frozen backbone, global softmax, and random rivals, each run over three seeds on CUB and Aircraft with both backbones. If the full model does not beat the frozen-backbone variant by more than seed variance on the confusable-pair metric, end-to-end training through the neighbor selection is not earning its cost, and we would say so.