When a classifier is trained on data where a few classes have thousands of examples and many have only a handful, the rare classes fail partly because their few images cover so little of the ordinary variation in lighting, color, background clutter and surface texture. The common classes show this variation thousands of times over. A large share of it is unrelated to class identity: a dimly lit photo of a heron and a dimly lit photo of a dog vary in much the same way. We propose to copy that variation from common classes onto rare ones. We work in the patch tokens of a frozen pre-trained Vision Transformer, the per-region feature vectors such a model produces for each small square of the image. The copied quantity is the style statistic from neural style transfer: the mean and the channel-by-channel correlation matrix of the tokens, also called the Gram matrix, which records which features fire together anywhere in the image and ignores where they fire.
The core claim has two parts. First, re-coloring a rare-class image's patch tokens with the second-order channel statistics of a common-class image produces new, label-preserving training examples for rare classes at almost no cost, with no image generator and no backbone fine-tuning. Second, the transfer must be gated. It should touch only the channel directions whose correlation statistics vary within classes and not between them, because in fine-grained domains texture is often exactly what defines a class. The combination of patch-level, second-order, gated transfer on frozen foundation-model tokens is what we believe is new. The decisive test is whether it improves few-shot-class accuracy over the strongest lightweight long-tail method and over a simpler transfer of global feature covariance.
What prior work establishes and leaves open
Three results motivate working on top of a frozen backbone. Parameter-Efficient Long-Tailed Recognition (LIFT) shows that fully fine-tuning CLIP on long-tailed data distorts its feature space and hurts tail classes. Its remedy is to tune about 0.1% of parameters through adapters, and combine this with text-embedding classifier initialization, a logit-adjusted loss (a loss that shifts decision thresholds by class frequency) and test-time ensembling. LIFT never looks at the statistics of the patch tokens. Rethink Long-tailed Recognition with Vision Transforms shows that Vision Transformers trained from scratch on imbalanced data collapse, reaching 38.66% versus 69.11% with masked-autoencoder pre-training. DeiT-LT repairs from-scratch training with a distilled CNN teacher. Together these papers constrain the design: any tail remedy should leave the pre-trained representation intact. TwinTURBO supplies the head architecture we reuse. It trains a single-layer transformer that reads frozen patch tokens plus the summary token, but only under balanced low-label splits and without touching channel statistics.
The method itself comes from the style literature. Image Style Transfer Using Convolutional Neural Networks and Texture Modelling Using Convolutional Neural Networks define style as Gram matrices of CNN feature maps. StyleLess layer shows that suppressing Gram correlations separates weather and lens nuisance from scene content (rain Cityscapes mIoU 55.2% to 69.9%). Both work only in trainable CNNs and never consider imbalance. We use the same statistic but move it between images rather than suppressing it, and we apply it to frozen transformer tokens.
Two results constrain the design directly. Style Augmentation found that blind style randomization hurts on ImageNet, where texture correlates with class; this is why our transfer is gated. Long Tail Image Generation Through Feature Space Augmentation fused head-class backgrounds into tail objects with diffusion and saw no tail mAP gain because of label noise. That failure is why we transfer only spatially pooled second-order statistics and never head-class spatial content.
The closest existing work is Risk-Bounded Distribution Reconstruction (RBDR). It reconstructs tail class means and covariances over whole-image pooled features, offline, and shrinks covariance along directions that point toward rival classes. Our proposal differs in three ways. It operates within each image across patch tokens rather than across images. It generates new token sets rather than calibrating a Gaussian. And it runs online while the head trains. Compositional Knowledge Transfer and CCL-Net also borrow from common classes, but they act on classifier weights and soft labels respectively, not on features.
Core Mechanism
We extract the patch tokens $X \in \mathbb{R}^{N \times d}$ of a frozen ViT for each image, where $N$ is the number of patches and $d$ the token width. We then project them onto a fixed principal-component basis of rank $r$ (around 64) so the correlation matrices stay small. For each image we compute the token mean $\mu$ and the covariance $\Sigma$, which is the centered Gram matrix.
On the common ("head") classes, where data is abundant, we split the $r$ directions into two sets. For each direction we compute a Fisher ratio of its per-image style statistics: the between-class variance of that direction's per-image variance and covariance entries, divided by their within-class variance. Directions with a low ratio form the nuisance subspace, with projector $P_n$. The rest form the discriminative subspace, with projector $P_d = I - P_n$.
To augment a tail image with tokens $X_t$, we draw a head-class donor image with statistics $(\mu_h, \Sigma_h)$ and recolor only the nuisance part:
$$X' = P_d X_t + \left[ (X_t - \mu_t)\, \Sigma_t^{-1/2} \Sigma_h^{1/2} + \mu_h \right] P_n,$$
where $\mu_t, \Sigma_t$ are the tail image's own statistics restricted to the nuisance subspace. The bracket is the whitening-coloring step of feature-level style transfer. The patch layout and the tail image's discriminative channels are untouched.
The augmented tokens train the TwinTURBO-style attention head with a logit-adjusted loss. We also test a second variant that applies the transfer at an intermediate block and lets the remaining frozen self-attention blocks re-integrate the recolored tokens. This costs extra forward passes but no backbone gradients. Donors can be drawn uniformly or from classes that are semantically close to the tail class, the latter echoing CCL-Net.
Research Program
The work proceeds in three stages. The first is diagnostic. We emulate tail data on head classes by subsampling some ImageNet-LT and Places-LT head classes to 5–20 images. On those classes we test whether test errors concentrate on images whose token style statistics lie far from every training image of their class, while holding the distance of the summary-token embedding fixed. If they do not, frozen foundation features are already style-invariant enough, and the mechanism has little room to help. We would report that negative result and stop.
The second stage builds the gate and validates it against held-out head classes. The question is whether low-Fisher directions transferred between head images ever flip a strong linear probe's prediction. This measures label preservation directly. The third stage is the full long-tail evaluation and the ablations described below.
Baselines
The naive reference is a linear probe and the same single-layer attention head on frozen CLIP ViT-B/16 tokens, trained with plain cross-entropy and then with logit-adjusted loss. This tells us how much of any gain is simply due to rebalancing the loss.
The strongest existing method is LIFT (Parameter-Efficient Long-Tailed Recognition), which reports 78.3% overall accuracy on ImageNet-LT with 0.62M learnable parameters, 80.4% on iNaturalist 2018, and 52.2% on Places-LT. We will rerun it from its released code on the same backbone, same epochs (10–20) and same hyperparameter search budget as our method, and report the reproduced figures alongside the published ones. Because our augmentation is independent of where the trainable parameters sit, we also run LIFT with our transfer added. Beating LIFT means a few-shot-split gain of at least two points over the reproduced LIFT, averaged over three seeds, with overall accuracy not lower.
The second strong competitor is global-statistic transfer in the spirit of RBDR. We borrow head-class within-class covariance of pooled embeddings, sample synthetic tail features from a Gaussian, and train the same head. RBDR reports 53.87% on CIFAR-100-LT at imbalance factor 100, against a 38.30% cross-entropy baseline, with a different backbone, so we rerun the approach rather than compare numbers across setups. For ISIC 2018 the reference is CCL-Net at 85.02% accuracy.
Three ablations isolate the key ingredient. Ungated transfer, with $P_n = I$, tests whether the gate matters. First-order-only transfer, which moves means and variances but not cross-channel correlations, tests whether the Gram matrix specifically matters. Random-subspace gating with the same rank tests whether the Fisher criterion selects the right directions. We also run RIDE (Long-tailed Recognition by Routing Diverse Distribution-Aware Experts, code at RIDE-LongTailRecognition) on the same features as a variance-reduction reference.
Why not just…?
Why not just use LIFT, which already wins with frozen features? LIFT's gains come from where it puts trainable parameters and from loss and initialization choices. It adds no information about the variation tail classes have never seen. Our transfer is complementary, so the fair test is LIFT with and without it. If the augmentation adds nothing on top of LIFT across all three large benchmarks, the proposal fails, and we will say so.
Why not just transfer covariance of the pooled embedding, as RBDR-style calibration does? This is the real threat. It is simpler and could plausibly match us. The argument for the patch-level version is that pooled-feature Gaussians mix nuisance and content in one covariance. RBDR itself needs risk bounds to stop tail samples from drifting into rival classes. Our transfer keeps every tail patch's spatial content and discriminative channels fixed by construction. The head-to-head comparison at matched synthetic-sample counts settles this. We will also measure rival confusion, meaning the rate at which tail test images are predicted as their nearest head class, because that is where we predict the difference appears.
Why not just style-augment the pixels and re-extract features? Image-level style transfer, as in Style Augmentation or patch-wise Random Style Replacement, needs a generator pass and a backbone forward pass per sample. It is also ungated, and it was reported to hurt where texture defines class. We include AdaIN-style pixel augmentation with matched compute as a baseline. iNaturalist, where species are distinguished by texture, is where we predict the ungated pixel version hurts and the gated token version does not.
Why not just use second-order pooled features in a linear probe? Feeding $\Sigma$ to the classifier makes style more available for classification. It does nothing to fill the tail's missing variation, and it may increase overfitting to the few tail styles. We include it to separate "style as a feature" from "style as transferable nuisance."
Proposed Experiments
All runs use frozen CLIP ViT-B/16, matching LIFT, and a DINO-family backbone as a second check. The benchmarks are ImageNet-LT, iNaturalist 2018, Places-LT and CIFAR-100-LT at imbalance factors 10 and 100. ISIC 2018 tests a domain far from web pretraining. We report overall accuracy and many-, medium- and few-shot splits, plus the predictive distribution calibration score from Rethink Long-tailed Recognition to show whether gains come from debiasing or from better tail features.
The checks run in order of how decisively they could refute the idea:
- Diagnostic. Does tail-emulated error concentrate on style-shifted test images? If not, the proposal stops.
- Gate prediction. We predict the ungated transfer helps on Places-LT, where scene style is mostly nuisance, but hurts on iNaturalist 2018. The gated version should help on both. If gating does not reverse the iNaturalist result, our account of the mechanism is wrong.
- Gram-specific contribution. Full second-order transfer must beat first-order transfer. If it does not, the Gram matrix is not the active ingredient.
- Main result. The method must beat global covariance transfer, and LIFT plus our transfer must beat LIFT, both on the few-shot split.
The intermediate-block injection variant is reported separately with its extra compute stated. That way any accuracy gain can be weighed against the additional forward passes.
References
- Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts
- Rethink Long-Tailed Recognition with Vision Transforms
- DeiT-LT: Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets
- TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spaces
- Image Style Transfer Using Convolutional Neural Networks
- Texture Modelling Using Convolutional Neural Networks
- StyleLess layer: Improving robustness for real-world driving
- Style Augmentation: Data Augmentation via Style Randomization
- Long Tail Image Generation Through Feature Space Augmentation and Iterated Learning
- Risk-Bounded Distribution Reconstruction: Stable Statistic Calibration for Long-Tailed Recognition
- Long-tail Recognition via Compositional Knowledge Transfer
- Learning and Exploiting Interclass Visual Correlations for Medical Image Classification
- Long-tailed Recognition by Routing Diverse Distribution-Aware Experts
- Data Augmentation Through Random Style Replacement