Balancing by Attributes, Not Classes, When Adapting Frozen Backbones to Long-Tailed Data

When a classifier is trained on data where a few common classes have thousands of images and many rare classes have a handful, the rare classes lose out. The usual fixes count images per class. They show rare-class images more often or weight their loss more heavily. This proposal starts from a different observation. Rare classes are short on images, but they are not short on visual evidence for the properties that define them. A rare warbler's "yellow throat" or "wing bars" also appear in hundreds of images of common birds. If the model learns features that detect those shared properties, called attributes, the rare class can borrow training signal from common-class images that it would never get from its own five photos. We propose to adapt a frozen pre-trained image model through a small trainable module. The module is trained jointly with a projection onto human-annotated attribute vectors, and the loss is balanced by how often each attribute is seen in training rather than how often each class is seen.

The question also settles a disagreement in the literature. Distribution Alignment retrains only the final classifier on balanced data and finds that this nearly closes the head–tail gap. It concludes that long-tail failure is a decision-boundary problem, not a feature problem. Zero-shot work such as DSSC and CSAP-ZSL finds the opposite for fine-grained semantics: frozen features lack the part-level detail needed to ground attributes. Our design separates the two. If attribute-balanced adaptation beats a balanced-classifier upper bound on the rarest classes, features are a real bottleneck for the tail. If it does not, Distribution Alignment's view extends to the semantic setting, and frozen features plus calibration are enough.

Core Claim

Under class imbalance, class-level rebalancing pushes the gradient toward the wrong images. It upweights a few tail images, which encourages memorizing them. Attribute-level balancing upweights any image, head or tail, that carries attributes which are rare in the training stream. These are mostly attributes that define tail classes.

We claim three things. First, an adapter trained on top of a frozen backbone with an attribute-alignment loss weighted this way improves Few-shot accuracy over class-balanced joint training. Second, it improves over frozen-feature methods with logit calibration. Third, it does so without the head-class collapse that LT-Soups documents for full fine-tuning on long-tailed data.

The mechanism is specific and testable. For attribute $a$, let $s_{c,a}\in[0,1]$ be its annotated strength in class $c$ and $N_c$ the number of training images of class $c$. The attribute's effective count is

$$n_a = \sum_c N_c\, s_{c,a}.$$

The per-attribute alignment loss is weighted by $w_a = n_a^{-\beta}$, with $\beta\in[0,1]$ moving from no reweighting to full inverse frequency. This mirrors the role of the $\rho$ knob in Distribution Alignment's generalized re-weighting, applied at the attribute level instead of the class level. For an image of class $c$, the alignment loss is

$$\mathcal{L}_{\text{attr}}(x) = \sum_a w_a\, \mathrm{BCE}\big(\sigma(g_a(h(\phi(x)))),\, s_{c,a}\big).$$

In this loss, $\phi$ is the frozen backbone, $h$ is the trainable adapter, $g_a$ is the projection to attribute $a$, and BCE is binary cross-entropy. The final class score is the compatibility between predicted attributes and each class's attribute vector, plus a standard logit-adjusted linear head. Training uses instance-balanced sampling, so head images, which are the source of transferable attribute evidence, are not discarded.

Relation to Prior Work

Two papers motivate the gap. DRAGON shows that class descriptors help long-tail recognition and introduces CUB-LT, SUN-LT and AWA-LT. However, its visual expert is trained separately and never adapts to the attribute space, and the authors leave shared "mid-fusion" representation learning open. Our method is exactly that kind of shared representation. SCILM addresses imbalance among seen classes in zero-shot learning with class-balanced batches over pre-extracted features. We keep instance sampling and move the balancing to attributes. SCILM's class-balanced batching becomes one of our central baselines, not our method.

Two papers constrain the design. LT-Soups shows that full fine-tuning on long-tailed data hurts tail classes while parameter-efficient tuning protects them. That result is why we train only an adapter. LT-Soups itself uses no semantic signal. Lin et al. show that frozen giant backbones beat full fine-tuning when data is scarce, which again argues for freezing. Decoupled Optimisation finds that head classes monopolize parameters inside the feature extractor. That cuts against Distribution Alignment, and our experiment adjudicates between them.

Other work supplies techniques or related constraints. CSAP-ZSL and DSSC supply the attribute-alignment objectives. DSSC's attribute-distance-adaptive triplet margin is an optional add-on in our method, but we avoid CSAP-ZSL's costly graph-cut region partitioning. VL-LTR uses Wikipedia text with CLIP distillation in two stages. We rely on compact attribute annotations instead, which also matters for domains without encyclopedic text. CALA points out that imbalance also exists within classes, across attributes. Our attribute counts address frequency across classes, not within-class attribute shift. We therefore treat CALA's setting as a stress test rather than a claimed solution.

Research Program

We first build the counting machinery: $n_a$ for each benchmark and an "attribute ownership" score for each attribute. The ownership score is the fraction of its mass that comes from tail classes. It lets us predict in advance which tail classes should benefit, namely those whose defining attributes are well represented in head images. This yields a falsifiable per-class prediction, not just an aggregate number.

Next we train the adapter-plus-projection model on frozen backbones of two strengths: an ImageNet-supervised ResNet-101, so results are comparable to the DRAGON benchmarks, and a large frozen Vision Transformer. Lin et al. report that larger frozen models carry more "ready-to-use" features, which may shrink the benefit of adaptation; testing at both scales checks that.

Finally we analyze where the gains come from. We measure attribute-prediction AUC separately for tail-owned and head-owned attributes. We also check whether Few-shot accuracy gains correlate with how much of a tail class's attribute mass is supplied by head images.

Baselines

The naive reference point is a frozen backbone with a linear head trained by plain cross-entropy on the long-tailed training set. DRAGON reports this kind of standard cross-entropy model at 53.0% on CUB-LT (DRAGON). We re-run it on our backbones so all methods share the same features.

The strongest descriptor-based method in the file is DRAGON itself, at 60.1% on CUB-LT and 51.2% per-class accuracy on ImageNet-LT-d (DRAGON). No reported result was found on Lacuna for DRAGON on SUN-LT or AWA-LT. We re-implement DRAGON's visual and semantic experts and its debiasing fusion on the same frozen backbone. Without that, a backbone upgrade would masquerade as a method gain.

The baseline we consider the real threat is frozen features plus an attribute-compatibility classifier plus logit adjustment and Distribution Alignment-style calibration, with no feature adaptation at all. Alongside it we run Distribution Alignment's "empirical classifier bound": a classifier retrained on a class-balanced version of the training set. This is an oracle-flavored upper limit for any method that leaves features fixed.

The remaining comparisons are SCILM-style class-balanced batching with the same adapter, and an LT-Soups-style recipe of a LoRA-tuned backbone followed by logit-adjusted classifier retraining. The key-ingredient ablation swaps attribute-level weights for class-level weights while holding everything else fixed. A second control replaces true attribute vectors with class-permuted ones, so the weights have the right marginal counts but the wrong semantics.

All methods share data splits, backbone, augmentation, and an equal hyperparameter search budget of the same number of trials per method, tuned on a held-out balanced validation split. Results are averaged over three seeds. Beating a baseline means improving Few-shot accuracy by more than two seed standard deviations while losing less than one point on Many-shot accuracy. Beating the classifier bound on Few-shot is the result that would show features, not only boundaries, limit the tail.

Why not just…?

Why not just use frozen features with an attribute classifier and logit adjustment? This is a real threat. Distribution Alignment's finding that balanced classifier retraining nearly closes the gap suggests it may suffice. CSAP-ZSL's evidence that off-the-shelf features miss fine part-level attributes suggests it will not on fine-grained data like CUB. The direct comparison against the classifier bound settles it. If the frozen calibrated model matches attribute-balanced adaptation on Few-shot, we will report that features are not the bottleneck and recommend the cheaper method.

Why not just balance by class while jointly training? That is SCILM's recipe. It upweights the few tail images, which risks memorizing them, the failure mode that the file attributes to naive oversampling (Towards Automated Sexual Violence Report Tracking). SCILM itself reports drops on dominant seen classes. Our class-weight versus attribute-weight ablation isolates exactly this difference.

Why not just LoRA-tune the backbone with logit adjustment, as LT-Soups does? Parameter-efficient tuning already protects the tail without any semantics, and LT-Soups reports 83.5% on CIFAR100-LT this way. But nothing in that objective directs the tunable capacity toward features shared with the tail. The permuted-attribute control tests whether semantic content, rather than extra capacity, drives our gains.

Why not just use a larger frozen backbone? Lin et al. show larger models need less adaptation, so the benefit may vanish at scale. Running both backbone sizes measures whether the gain shrinks. A shrinking gain would be a meaningful, reportable result.

Proposed Experiments

The main experiments run on CUB-LT, SUN-LT and AWA-LT from DRAGON. These benchmarks have real attribute vectors, so $n_a$ is well defined. We report per-class accuracy and Many/Medium/Few splits.

ImageNet-LT-d is a secondary test. Its Word2Vec descriptors are not interpretable attributes, so we either use them directly or compute effective counts over embedding dimensions. We expect weaker effects there, and say so in advance.

For transfer, we run generalized zero-shot evaluation on CUB and aPY with artificially long-tailed seen classes, scored by harmonic mean. This tests whether attribute-balanced features also help unseen classes that depend on rare seen ones, which is SCILM's motivating case. As a stress test, we evaluate on ImageNet-GLT, where CALA's within-class attribute shift may break the assumption that attribute evidence transfers cleanly.

Three checks decide the claim. The first is whether attribute-balanced adaptation exceeds the frozen classifier bound on Few-shot accuracy. The second is whether it beats class-balanced weighting under identical training. The third is whether per-class gains track the predicted head-supplied attribute mass rather than appearing uniformly. If all three hold, the proposal's mechanism is supported. If the first fails, the honest conclusion is that semantics belong in the classifier, not the features.