How Much Similarity Can Sibling Classes Share? Leaf-Preserving Soft Contrastive Targets for Fine-Grained Linear Probing

Most contrastive training treats similarity as all-or-nothing. Two images either count as "the same" and are pulled together, or count as "different" and are pushed apart equally hard. For fine-grained recognition this is a poor fit. Two warbler species are far more alike than a warbler and a truck, yet a binary loss repels them just as strongly. The obvious fix is to give related pairs partial similarity targets, using a teacher model's distances or a label taxonomy. But this fix carries a hidden risk.

In fine-grained data the most related pairs are exactly the sibling classes a downstream classifier must tell apart. Pulling them together could erase the directions that separate them. We propose to measure this risk under linear probing, meaning a single linear classifier trained on frozen features, which tests whether the classes are separable by flat boundaries. We also propose a fix for it. Our claim is that soft similarity helps fine-grained linear probing only when every leaf class keeps a private share of the target similarity that no sibling can claim. That share can be specified exactly, measured in the learned features, and used to predict when relaxation helps and when it hurts.

The question sits on an unresolved tension in the literature. Methods that relax binary targets report better generalization, but they report it almost entirely through nearest-neighbour retrieval. Embedding Transfer With Label Relaxation for Improved Metric Learning distils Gaussian-kernel similarities from a teacher and evaluates by Recall@K on CUB-200-2011, Cars-196 and Stanford Online Products. CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss evaluates its soft image-text targets zero-shot on coarse benchmarks. Meanwhile, linear-probing audits such as Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation show frozen features collapsing from coarse to fine ranks, from 99.7% at the "Class" level to 61.1% at the species level on CUB for InternVL3. Nobody has checked whether relaxing similarity widens that gap or closes it.

Core Claim

Assume a label taxonomy with ranks $r = 1, \dots, L$, where rank $L$ is the leaf (species, or make-model-year). Assign each rank a weight $\alpha_r \ge 0$, with the weights summing to 1. For two images from leaf classes $a$ and $b$, let $\ell(a,b)$ be the depth of their lowest common ancestor. We set their target cosine similarity to

$$s(a,b) = \sum_{r=1}^{\ell(a,b)} \alpha_r ,$$

so same-class pairs get target 1, and siblings get $1-\alpha_L$. The weight $\alpha_L$ is the leaf share: the part of each class's identity that no other class shares.

This target matrix is exactly the similarity matrix of a construction we call a nested simplex. Each class mean is a sum of mutually orthogonal vectors, one per ancestor, where the vector for rank $r$ has squared length $\alpha_r$. Two consequences follow, both by construction. First, the target is always a valid similarity matrix. Second, any $\alpha_L > 0$ leaves every class a private orthogonal direction, so an encoder that fits the target perfectly is linearly separable at every rank. Setting $\alpha_L = 1$ recovers ordinary binary supervised contrast. Teacher-kernel targets of the kind used by Kim et al. carry no such guarantee: two near-identical sibling species can receive targets near 1, which leaves them no private direction at all.

The proposal makes three testable claims. The first is that species-level linear-probe accuracy is non-monotone in $\alpha_L$: moderate shared weight beats binary training, and too little leaf share falls below it. The second is that the shared ranks mainly buy robustness, showing up as gains on uncurated data and at coarse ranks rather than on the curated test set. The third is that, across every objective we train, probe accuracy is predicted by a measured version of the leaf share in the learned features, and the name of the loss adds little once that is known.

For the third claim we need a measured quantity. We define sibling-residual energy: for each class, take its feature mean, remove the component lying in the span of its siblings' means, and compute the fraction of squared norm that remains. If relaxed similarity helps when this quantity stays high and hurts when it is driven low, the linear-separability tension is resolved and given a usable dial.

Research Program

The work has three parts. We first build the objective. The nested target $s(a,b)$ feeds a continuously weighted contrastive loss in the style of CWCL. In this loss, a batch image's softmax distribution over the other images is pulled toward the normalized soft targets rather than toward a single positive. We keep instance-level positives (two augmented views of one image) at target 1. Kim et al. found that relaxed losses resist overfitting on held-out classes, and we reuse that idea, but our targets come from taxonomy weights we choose rather than from a teacher. This avoids the teacher error cascade those methods inherit.

We then build the diagnostic. Every trained encoder is frozen and probed with a linear classifier at each taxonomic rank. For each model we compute sibling-residual energy, k-nearest-neighbour recall, and the probe accuracies. This lets us check directly whether kNN gains and probe gains diverge, which is the scenario the tension predicts.

Finally we connect the results to adaptation. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution showed that probing first and then fine-tuning (LP-FT) moves features 10–100× less than direct fine-tuning, and gains about 10% out-of-distribution. We run LP-FT on top of each encoder to test whether a well-conditioned nested geometry also resists distortion during adaptation.

The taxonomies are concrete. For CUB we use order, family, genus and species, matching the ranks FG-BMK probes. For Stanford Cars we use make, model and year. For FGVC-Aircraft we use manufacturer, family and variant. Every encoder starts from the same pretrained ViT-B backbone. We use both a CLIP initialization and a DINOv2 initialization, because FG-BMK reports that contrastive backbones of this kind probe best on fine-grained tasks. Each encoder is then trained on the fine-grained training split with one of the competing objectives.

Baselines

The do-nothing reference is the pretrained CLIP or DINOv2 backbone probed directly, with no further training. This tells us whether any objective adds linear separability beyond what the backbone already has.

The binary reference is supervised contrastive training with leaf labels only, which is our objective at $\alpha_L = 1$. It shares our code, augmentations, epochs and tuning budget, so any difference between the two isolates the shared-rank weights.

The strongest existing relaxed method is the relaxed contrastive loss of Embedding Transfer With Label Relaxation for Improved Metric Learning. We run it with the do-nothing backbone as teacher and tune the kernel bandwidth over the same grid size we use for $\alpha$. Kim et al. report Recall@1 only in a bar chart, and no reported numeric result was found on Lacuna for it. We will therefore reproduce it rather than quote it.

The strongest taxonomy-aware precedent is the softmax-plus-generalized-triplet model of Embedding Label Structures for Fine-Grained Feature Representation. It reports 88.4% classification accuracy on Stanford Cars, trained end-to-end rather than probed. We reimplement its hierarchical quadruplet loss on our backbone and probe it frozen. We also cite that 88.4% as the end-to-end reference that a probed model must approach.

We add taxonomy-smoothed cross-entropy, described in the next section, and an ANCOR-style angular contrast following Fine-Grained Angular Contrastive Learning With Coarse Labels for the coarse-label condition.

The key ablation sweeps $\alpha_L$ from near 0 to 1 while holding everything else fixed. It also includes a shuffled-taxonomy control: the same distribution of soft targets, but assigned to random class pairs.

All methods get equal GPU-hours and an equal number of hyperparameter trials. All are probed with the same L2-regularized logistic regression, with the regularization strength chosen on a validation split. We count it as beating the baselines if the best $\alpha_L$ improves species-level probe accuracy over both binary SupCon and teacher-relaxed contrast by more than the across-seed standard deviation (three seeds) on at least two of the three datasets, and does not lose on coarse ranks.

Why not just…?

Why not just use taxonomy-aware label smoothing with cross-entropy? This is the most serious threat. Smoothing soft labels in proportion to tree similarity is simpler and may induce a similar geometry. We do not argue it away. We include it, tuning its smoothing mass and per-rank split on the same budget, and we compute its sibling-residual energy. If it matches our objective on probe accuracy, the contribution narrows to the diagnostic: the claim that leaf share, not the loss family, determines fine-grained linear separability. That claim is still testable, and it is still new. Uniform label smoothing, by contrast, treats a truck and a sibling species alike, and is included only as a weaker reference.

Why not just keep binary contrast and add stronger augmentation? CWCL's gains (74.41% vs. 66.84% zero-shot ImageNet over binary locked-image tuning) come from fixing the targets themselves, not from changing the inputs. Augmentation does not stop a binary loss from repelling siblings at full strength. Still, the binary SupCon baseline uses our strongest augmentation recipe. If it matches the best $\alpha_L$, our first claim fails.

Why not just use an MLP probe or kNN instead of a linear probe? A nonlinear probe can build the missing features itself, so it hides exactly the deficit we are measuring. kNN is reported anyway. Showing that kNN and linear probing disagree is one of the outcomes we test for, not something to avoid.

Proposed Experiments

The first decisive check is the $\alpha_L$ sweep on CUB, Stanford Cars and FGVC-Aircraft. We plot species-level and coarse-rank probe accuracy, along with kNN Recall@1, against leaf share. The claim requires an interior optimum, plus a regime of low $\alpha_L$ where kNN holds steady while species probing drops.

The second check is robustness. We train on CUB and probe on iCub100 and on CUB++ labels. Facing the Hard Problems in FGVC reports that roughly 90% curated accuracy falls to about 50% on iCub100. It also reports that CUB contains roughly 4% label errors, and that correcting them lifts all methods by 2–3%. The prediction is that shared ranks shrink the curated-to-wild gap more than they raise curated accuracy.

The third check is cross-method prediction. Across all trained encoders, seeds and datasets, we compute the rank correlation between sibling-residual energy and species probe accuracy. We compare it against the correlation obtained from the loss label alone, and against a training-free transferability score of the kind proposed in LogME: Practical Assessment of Pre-trained Models for Transfer Learning. The diagnostic only earns its place if sibling-residual energy predicts probe accuracy better than these.

The fourth check is a coarse-label condition. Here supervision stops at genus or make, and species are probed. This tests whether the coarse "alignment tax" FG-BMK observes comes from binary coarse targets. If so, a nested target with an instance-level private component should preserve species directions better than binary coarse contrast or ANCOR.

Finally, we run LP-FT from each encoder and measure how far the features move. The prediction is that a high-leaf-share initialization moves least and keeps the largest iCub100 advantage.