When a large pretrained image model such as CLIP is fine-tuned on a dataset where a few classes have thousands of images and most have only a handful, the rare classes suffer most. Their internal representations drift toward the common classes, and the useful knowledge the model brought from pretraining is lost. One established fix holds the fine-tuned model's internal features close to those of the original model, a technique called feature-space regularization. Applying that pull with equal strength to every class is a mistake in this setting. For a rare class the pretrained model already understands well, the pull protects knowledge that a few labeled examples could never rebuild. For a class that was also rare in the model's web-scale pretraining data, the same pull locks in a representation that is known to be incomplete and biased.
We propose to set the strength of the feature anchor separately for each class, using two quantities that can be computed before training starts. The first is how many downstream examples the class has. The second is how well the frozen pretrained features already separate that class, measured directly on the downstream training images. The anchor is strong where the pretrained features are trustworthy and data are scarce. It is relaxed where the pretrained features are unreliable, so that genuine feature learning can happen there. The method is a single weighted loss term added to a standard parameter-efficient fine-tuning recipe. Its value rests on one testable prediction: the benefit of anchoring depends on the interaction between upstream reliability and downstream frequency, not on frequency alone.
Core Claim
Feature preservation methods treat the pretrained representation as ground truth everywhere. LDIFS (Fine-tuning can cripple your foundation model; preserving features may be the solution) penalizes the squared distance between fine-tuned and original features across several layers. It shows this prevents forgetting better than weight-space penalties, but it applies the penalty uniformly and was not evaluated under label imbalance. REPINA (Representation Projection Invariance Mitigates Representation Collapse) relaxes the point-to-point penalty by allowing a learned map between the two feature spaces, again uniformly across examples. Both methods supply the mechanism we reuse.
Two findings constrain how that mechanism should be used under imbalance. Rethinking the Bias of Foundation Model under Long-tailed Distribution shows that CLIP-family models inherit "parameter imbalance" from skewed web data. Classes rare both upstream and downstream (which it calls Tail-Tail) stay poor even after logit adjustment, because the defect sits in the features. Uniform anchoring would preserve exactly that defect. Long-Tailed Learning Requires Feature Learning proves, in a synthetic model, that fixed features fail on rare concept combinations while networks that update their features succeed. Freezing, or anchoring too tightly, is therefore not a safe default for classes whose features are wrong.
The closest work on the imbalance side, Inducing Neural Collapse in Deep Long-tailed Learning, weights a within-class compactness loss inversely by class frequency. It trains CNNs without any pretrained anchor, and its weighting depends on downstream counts only. Our claim is that downstream frequency is the wrong sole axis once a pretrained model is involved. The anchor weight must also depend on whether the pretrained features for that class deserve preserving.
Concretely, let $f_\theta$ be the model being fine-tuned and $f_0$ the frozen original model, with features taken from a small set of layers $\ell \in L$. Let $n_c$ be the number of training images of class $c$, and $n_{\max}$ the largest such count. Let $r_c \in [0,1]$ be the leave-one-out nearest-neighbour accuracy of class $c$ computed with frozen $f_0$ features on the downstream training set. The added loss is
$$ \mathcal{R}(\theta) = \frac{1}{|B|}\sum_{(x,y)\in B} \lambda_y \sum_{\ell \in L} \big\| f^{\ell}_\theta(x) - f^{\ell}_0(x) \big\|^2, \qquad \lambda_c = \lambda\, r_c^{\beta} \Big(\frac{n_{\max}}{n_c}\Big)^{\alpha}, $$
Here $B$ is a minibatch and $\lambda$ is a global strength. The exponents $\alpha$ and $\beta$ control how much frequency and reliability each modulate the weight, and the $\lambda_c$ are renormalized so their frequency-weighted mean equals $\lambda$. Setting $\beta=0$ recovers frequency-only weighting. Setting both exponents to zero recovers uniform LDIFS. Both settings are the ablations that test the claim. Measuring reliability on the downstream images costs one forward pass of the frozen model and needs no access to pretraining data.
Research Program
The base recipe follows LIFT (Parameter-Efficient Long-Tailed Recognition). LIFT updates under 1% of CLIP's parameters through low-rank adapters, initializes the classifier from text embeddings of the class names, and trains with a logit-adjusted loss. We add $\mathcal{R}$ on top. Because LIFT already restricts drift through which parameters move, the first question is whether explicit, class-weighted feature anchoring adds anything beyond that implicit constraint. The evidence in Semantic-guided Fine-tuning of Foundation Model for Long-tailed Visual Recognition suggests it can: that paper finds attention on tail classes still drifts to background under parameter-efficient tuning. Sage counters this with text-conditioned adapters in every block and a logit-level merge with frozen CLIP. Our approach instead regularizes intermediate features directly and adds no architecture.
The program has three parts. The first is diagnostic. We bin classes by downstream frequency (the standard many, medium and few splits) and by upstream reliability $r_c$. We then measure, per bin, how far features drift from $f_0$ under plain fine-tuning, LIFT, and uniform LDIFS, together with per-class accuracy and the effective rank of class-conditional features, following the rank-collapse framing of REPINA. This tests whether the harm of drift and the harm of anchoring really separate along the reliability axis.
The second part is the method and its ablations as defined above. We also run a variant that replaces the point-wise anchor with a REPINA-style identity-plus-shift map for low-reliability classes. This asks whether "released" classes do better with geometry-preserving freedom than with simply a weaker pull.
The third part tests the proxy. Where an upstream frequency split in the sense of Rethinking the Bias of Foundation Model under Long-tailed Distribution can be obtained or reconstructed, we check that low $r_c$ actually tracks Tail-Tail membership. We also check whether $r_c$ predicts which classes gain from relaxation better than upstream frequency does.
Baselines
The naive reference is full fine-tuning of CLIP ViT-B/16 with cross-entropy, plus the zero-shot frozen model. Together these bracket the cost of drift and the value of doing nothing.
The strongest single-model competitor is LIFT, because it shares our backbone, parameter budget and loss. Any gain over it must come from the anchoring term. LIFT reports 78.3% top-1 on ImageNet-LT, 80.4% on iNaturalist 2018 and 52.2% on Places-LT (Parameter-Efficient Long-Tailed Recognition). We will rerun it with its released configuration and the same seeds, epochs and augmentation as our method. Beating it means a higher overall top-1 averaged over three seeds with few-shot accuracy not lower, and especially a few-shot gain larger than seed variance.
Sage is the strongest published method that also attacks tail drift in CLIP. It reports 83.3% on CIFAR-100-LT with imbalance ratio 100, 1.6 points over LIFT, and 53.8% tail accuracy on Places-LT (Semantic-guided Fine-tuning of Foundation Model for Long-tailed Visual Recognition). We will compare against its reported numbers. If code is available we will also rerun it under our budget, reporting its extra parameter count alongside accuracy.
The multi-model backdoor adjustment of Rethinking the Bias of Foundation Model under Long-tailed Distribution reports few-shot gains of 3.49 points over LIFT on ImageNet-LT and 1.91 points on iNaturalist. It is the reference for Tail-Tail repair. Matching a meaningful fraction of its few-shot gain with one model at a fraction of its inference cost would count as success; exceeding it is not required.
Uniform LDIFS layered on LIFT with logit adjustment is the essential ablation. No reported result found on Lacuna for LDIFS on any long-tailed benchmark, so we run it ourselves, tuning $\lambda$ over the same grid as our method.
Frequency-only weighting ($\beta=0$) is the second ablation and the obvious transfer of the Neural Collapse paper's inverse-frequency idea. All methods receive identical hyperparameter search budgets measured in GPU hours.
Why not just…?
Why not just use logit adjustment on top of LIFT, since it already handles imbalance? Logit adjustment moves decision thresholds but leaves features unchanged. Rethinking the Bias of Foundation Model under Long-tailed Distribution shows it fails to fix biased representations on Tail-Tail classes. Our method keeps logit adjustment and asks only whether feature-level control adds to it. The LIFT comparison settles this directly.
Why not just apply uniform LDIFS and tune one global strength? This is the most serious threat. A well-tuned uniform anchor could capture most of the benefit if reliability and frequency happen to be nearly redundant on these benchmarks. Our prediction is that uniform anchoring helps high-reliability few-shot classes but hurts low-reliability ones, so that the per-bin pattern shows a crossover. If uniform LDIFS matches our method in every reliability-by-frequency bin, the per-class weighting idea is wrong and we will report it as such.
Why not just weight by inverse frequency, as the Neural Collapse paper does? Frequency-only weighting anchors rare classes hardest, including Tail-Tail classes whose pretrained features are poor. The $\beta=0$ ablation isolates this. The decisive evidence is whether accuracy on low-$r_c$ few-shot classes improves when reliability enters the weight. A known confound is that $r_c$ correlates with frequency, so we report the partial correlation and the gains within each frequency bin.
Why not just ensemble several foundation models to deconfound upstream bias? That approach works, as its reported gains show, but it multiplies inference memory and compute by the number of models. Our question is how much of that repair one model can recover through training-time feature control. If ours falls far short on Tail-Tail classes, the honest conclusion is that some upstream defects cannot be fixed by regularization alone.
Proposed Experiments and Decisive Checks
We will evaluate on ImageNet-LT, iNaturalist 2018, Places-LT and CIFAR-100-LT with imbalance ratios 100 and 10, reporting overall top-1 and many/medium/few accuracy over three seeds. We additionally report accuracy on the 2-by-2 split of frequency against reliability (median split on $r_c$). Places-LT is the most informative arena, because scene categories are where pretrained CLIP features are least reliable and tail accuracy remains low.
The first decisive check is the interaction test. Relative to uniform LDIFS, our gain should concentrate in the low-reliability few-shot cell, with no loss in the high-reliability few-shot cell. A main effect of frequency alone would not support the claim.
The second check is a drift-accuracy curve per class across the $\lambda$ grid. Optimal drift should rise as $r_c$ falls. If optimal drift is flat across reliability, the mechanism is not what we claim.
The third check repeats training with REPINA-style relaxation for released classes and measures effective rank of tail-class features. This tells whether released classes need freedom to move or freedom to reshape.
Finally, we report the cost of $\mathcal{R}$ in memory and time, which comes from one extra forward pass through the frozen model. These costs are compared against LIFT and against the multi-model ensemble, so that any accuracy gain is weighed against what it costs to obtain.
References
- Fine-tuning can cripple your foundation model; preserving features may be the solution
- Representation Projection Invariance Mitigates Representation Collapse
- Rethinking the Bias of Foundation Model under Long-tailed Distribution
- Long-Tailed Learning Requires Feature Learning
- Inducing Neural Collapse in Deep Long-tailed Learning
- Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts
- Semantic-guided Fine-tuning of Foundation Model for Long-tailed Visual Recognition