Spending Attention Diversity Where the Data Lacks It: Diversity-Targeted Attention Regularization for Long-Tailed Vision Transformers

When a classifier is trained on long-tailed data, a few common classes supply most of the examples and many rare classes supply only a handful. The rare classes suffer in two ways. They have fewer examples, and the examples they do have often vary little: the same pose, the same background, the same lighting. A vision transformer is an image model that splits a picture into patches and lets each patch "attend" to, or weight, the others. Trained from scratch on such data, it can learn to recognize a rare class from one salient cue and ignore everything else. This failure is called attention collapse. We propose to measure, class by class, how collapsed the model's internal representation of each image is. A penalty on redundant attention would then be applied only where that collapse is worst, and switched off once a class reaches the internal diversity that well-sampled classes show naturally. In short, the model is asked to see more varied evidence in exactly the classes where the data failed to provide varied examples.

This matters most where the standard fixes are unavailable. Masked-autoencoder pre-training makes transformers largely insensitive to class imbalance, but it needs large unlabeled corpora. Distillation from a convolutional teacher works, but it needs a second, separately trained network. Diffusion-based tail augmentation needs a generative model trained from scratch. A single-network regularizer that is cheap and does its work during ordinary training would serve medical, private, and niche domains. The same experiments also test a scientific question that is useful whatever the outcome: can diversity in a model's internal attention partly substitute for diversity in its training data?

Core Claim

Our claim is that the tail-class failure of from-scratch vision transformers is partly an internal-diversity failure, and that it can be measured directly as excess redundancy among token attention patterns and low effective rank of patch features within each image. The second claim is that diversity pressure aimed by this measured deficit beats both uniform diversity pressure and pressure aimed by class frequency. The proposal differs from existing attention diversity work, which applies one global penalty to every sample. It also differs from long-tailed transformer methods, which create diversity only at the output level, between two tokens or between expert branches.

Our view of the closest prior work is as follows. Robustifying Token Attention for Vision Transformers supplies the method we build on. Its Attention Diversification Loss penalizes cosine similarity between the attention vectors of different tokens, and it improved corruption error by 2.4% at a cost of about 0.3M parameters, but it applies the penalty uniformly and was never tested under imbalance. DeiT-LT motivates the proposal. It shows that the distillation token, trained with deferred re-weighting against a CNN teacher, attends to local, fine-grained features and serves as a tail expert, while the classification token attends globally. That specialization is induced only at the two output tokens and requires a teacher; the intermediate heads are left unconstrained. GUIDE and ComReg diagnose "homogeneity collapse" across expert branches and counter it with prediction divergence or prototype contrastive learning. We ask whether diversity enforced inside one backbone can achieve part of that effect without branches or multi-pass inference. On the data side, Data-Centric Long-Tailed Image Recognition constrains our design. Its Feature Diversity Gain, a log-determinant measure of class feature volume, shows a "Goldilocks zone": moderate added diversity helps tail classes, while excessive diversity causes class overlap. This is why our penalty is capped and turns off at a target, rather than maximizing diversity without limit.

Research Program

The work has three parts: a diagnostic, a regularizer, and a controlled study of data diversity.

The diagnostic comes first, because the rest of the work only makes sense if collapse is real and concentrated in the tail. We train a plain ViT from scratch on CIFAR-100-LT and ImageNet-LT with balanced cross-entropy. For every image we record two quantities. The first is the mean pairwise cosine similarity among the token attention vectors in each layer and head, which is the quantity the Attention Diversification Loss penalizes. The second is the effective rank of the image's patch-feature matrix, defined as the exponential of the entropy of its normalized singular values.

The second measure must be computed per image, over its patches, rather than across a class's images. Across-image rank is capped by sample count: a class with five images can have rank at most five, which would wrongly mark every tail class as collapsed. We then plot both measures against class frequency for the many-shot, medium-shot, and few-shot splits. We also correlate them with per-class error, controlling for frequency.

The regularizer uses the diagnostic directly. Let $r_c$ be a running average of per-image effective rank for class $c$, and let $\bar r_{\text{head}}$ be its median over the many-shot classes. The training loss is

$$ \mathcal{L} = \mathcal{L}_{\text{BalCE}} + \lambda \sum_{c} w_c \frac{1}{|B_c|}\sum_{x \in B_c} \mathcal{L}_{\text{ADL}}(x), \qquad w_c = \mathrm{clip}\!\left(\frac{\bar r_{\text{head}} - r_c}{\bar r_{\text{head}}}, 0, 1\right). $$

Here $\mathcal{L}_{\text{BalCE}}$ is balanced cross-entropy, $B_c$ is the set of class-$c$ images in the minibatch, $\mathcal{L}_{\text{ADL}}$ is the thresholded token-attention similarity penalty of Robustifying Token Attention for Vision Transformers, and $\lambda$ is a single global strength. A class that already has head-level internal diversity receives no penalty. A collapsed class receives up to the full penalty.

The cap at head-level diversity is our answer to the Goldilocks finding. It stops the penalty from scattering tail attention onto background patches, the overcorrection that attention diversity papers warn about. In the main runs we use ground-truth labels to assign $w_c$ during training. A variant that assigns $w_c$ from the predicted class tests whether the targeting signal can be obtained without labels, which matters for later use in semi-supervised settings.

The controlled study separates data diversity from data quantity. On CIFAR-100-LT at imbalance factor 100, we hold each tail class's sample count fixed and change only which examples are kept. In a "spread" condition we keep examples chosen by k-center selection in the feature space of a reference network trained on balanced CIFAR-100. In a "clustered" condition we keep the examples nearest a single centroid. The reference network is used only to build these splits and never touches the trained models.

We quantify each condition's tail diversity with the log-determinant volume behind Feature Diversity Gain. Then we train each method under both conditions. If attention diversity substitutes for data diversity, the regularizer's gain should be clearly larger in the clustered condition. It should also recover a measurable fraction of the gap between clustered and spread training.

Baselines

The naive reference is a ViT trained from scratch with balanced cross-entropy, using the same architecture, augmentation, and epoch budget as our method. Rethink Long-tailed Recognition with Vision Transforms found balanced cross-entropy to be the most effective plain loss for long-tailed transformers. For context, from-scratch ViTs are weak: DeiT-LT reports 62.6% for vanilla ViT on CIFAR-10-LT at imbalance 100. Rethink Long-tailed Recognition with Vision Transforms reports 38.66% from scratch on ImageNet-LT. Any improvement must be shown against this reference first.

The strongest existing method on this exact question, from-scratch transformers without external data, is DeiT-LT. It reports 87.5% accuracy on CIFAR-10-LT at imbalance 100, 59.1% on ImageNet-LT against DeiT-III's 48.4%, and 75.1% on iNaturalist 2018. We use the DeiT-LT recipe, including Mixup and CutMix, for both conditions. Our method is evaluated in two forms: standalone, without the teacher, and added on top of DeiT-LT.

We count the regularizer as successful under either of two conditions. The standalone form must recover at least half of the gap between the balanced cross-entropy reference and DeiT-LT while using no teacher. Alternatively, the combined form must beat DeiT-LT's few-shot accuracy across three or more seeds, with overall accuracy no worse. We give DeiT-LT's teacher training time to our method as an extra tuning budget, so that compute is matched.

On CIFAR-100-LT, the relevant context numbers are 54.1% for ComReg and 52.4% for DiffuLT, both at imbalance 100. These methods use different backbones and extra machinery, so we report them as context rather than as matched competitors. We found no reported DeiT-LT result on CIFAR-100-LT on Lacuna, so we will rerun DeiT-LT ourselves there.

Two ablations isolate the key ingredient, which is the targeting. The first applies the same penalty uniformly to all classes ($w_c = 1$), as in the original Attention Diversification Loss. The second sets $w_c$ by inverse class frequency instead of by measured collapse. Both get the same $\lambda$ sweep as our method. A third ablation removes the cap and lets $w_c$ grow without bound, which tests whether the Goldilocks constraint matters inside the model as it does for data.

Why not just…?

Why not just apply the existing attention diversification loss uniformly together with balanced cross-entropy? This is the most serious threat. The uniform penalty may already help tail classes most, because tail images are the collapsed ones and so incur most of the penalty anyway. The uniform ablation tests this directly. If it matches the targeted version within seed noise on few-shot accuracy, we will drop the targeting claim. The contribution would then shrink to the diagnostic and the data-substitution study, which remain informative. A smaller targeted advantage could still appear on many-shot accuracy, where the uniform penalty may push well-sampled classes onto irrelevant patches. That pattern would support targeting.

Why not just weight the penalty by class frequency? Frequency is free to compute, while per-image effective rank costs a singular value decomposition of a small matrix. The controlled study is where the two signals come apart. In the spread condition, tail classes are rare but diverse, so frequency weighting would still apply heavy pressure and our measure would not. If frequency weighting wins in both conditions, measured collapse adds nothing and we will report that.

Why not just distill from a CNN teacher as DeiT-LT does? DeiT-LT already reaches 87.5% on CIFAR-10-LT, and we do not expect a regularizer alone to beat it. Our claim is narrower. We aim to recover much of that gain without a teacher, and to be complementary on few-shot classes when combined with distillation, since DeiT-LT leaves the intermediate heads unregularized. Both claims are tested in the baseline comparison.

Why not just pre-train with masked autoencoders? Fine-tuning a pre-trained masked autoencoder with balanced cross-entropy reaches 77.43% on iNaturalist 2018 and 69.11% on ImageNet-LT (Rethink Long-tailed Recognition with Vision Transforms). Where large unlabeled data exist, this likely dominates, and we will not claim otherwise. That paper itself notes that masked pre-training needs a sufficiently large dataset. Our target regime is therefore small, from-scratch settings such as ISIC 2018. As a secondary experiment, we will also add the regularizer during masked-autoencoder fine-tuning to see whether any residual tail collapse remains after pre-training.

Why not just generate more diverse tail data? DiffuLT raises ImageNet-LT few-shot accuracy from 5.8% to 39.9%, but it trains a diffusion model from scratch, which is far more costly than an auxiliary loss. The clustered-versus-spread study is our direct test of how much internal diversity can stand in for data diversity. If it recovers little of the gap, generation remains the better tool and the regularizer's role becomes complementary.

Proposed Experiments and Decisive Checks

The main benchmarks are CIFAR-10-LT and CIFAR-100-LT at imbalance factors 10, 50, and 100, ImageNet-LT, and iNaturalist 2018. On each we report overall top-1 accuracy and accuracy on the many-shot, medium-shot, and few-shot splits. ISIC 2018 serves as a small medical test, where CCL-Net reports 83.47% to 85.02% accuracy with ResNet-18. There we report accuracy and Cohen's kappa, a chance-corrected agreement score that is less inflated by imbalance than raw accuracy. Every result is averaged over three or more seeds and comes with the diagnostic curves: attention similarity and effective rank by class frequency, before and after training. For ImageNet-LT models we also report corruption error on ImageNet-C, since the base loss was designed for robustness.

Three results would decide the proposal.

  1. The diagnostic. If tail images in from-scratch transformers are not more collapsed than head images, controlling for frequency, the premise fails and we stop.
  2. The ablation against uniform and frequency weighting. The targeted version must beat both on few-shot accuracy without losing many-shot accuracy, or the targeting is unnecessary.
  3. The interaction in the controlled study. Regularizer gains should concentrate in the clustered condition, and they should vanish when the cap is removed and attention spreads onto background. Absent that interaction, the claim that attention diversity substitutes for data diversity is false, even if accuracy improves for other reasons.