A vision transformer splits its attention into several parallel "heads," each of which can look at different parts of an image. Pretraining with masked image modeling (MIM), which trains the model to fill in hidden image patches, is known to leave these heads looking at different places. Supervised and contrastive training instead tend to make the heads redundant, all fixating on the same salient region. Label-efficient learning, meaning adapting a pretrained model with only 1–10% of the usual annotations, then fine-tunes that model on a tiny labeled set. Nobody has checked what this small-data fine-tuning does to head diversity. Our hypothesis is that strong, noisy gradients from a few hundred labels push heads toward the single most label-predictive region, so a pretrained model with diverse heads is quietly turned into an effectively single-head one. If that is true, part of the value of pretraining is thrown away during adaptation, and the losses should be largest where multiple regions matter: rare findings, fine-grained categories, and segmentation.
We propose to measure this merging directly and then test a simple remedy. The remedy is a constraint that lets each head change freely during fine-tuning but penalizes any pair of heads for becoming more similar than they were after pretraining. The point of the design is to separate two explanations that current methods blur together. Fine-tuning might hurt because heads drift away from their pretrained behavior, which would call for anchoring them. Or it might hurt because heads collapse onto each other, which would call for keeping them apart. The experiments are built so that either answer is informative.
Core Claim
In few-label fine-tuning of vision transformers, the damaging change is heads merging, not heads changing. A hinge penalty on increases in pairwise head similarity, measured relative to the pretrained model, should recover accuracy at 1% labels that standard fine-tuning loses. It should also beat both anchoring each head to its pretrained attention and generic diversity losses, with the gains concentrated on rare classes and dense prediction. Because the target similarity comes from the pretrained model itself, the penalty needs no hand-set diversity level. That addresses the known hyperparameter brittleness of absolute diversity losses.
Relation to Prior Work
Two papers motivate the hypothesis without testing it. A Theoretical Analysis of Self-Supervised Learning for Vision Transformers proves, for a one-layer softmax ViT, that contrastive objectives send attention to dominant global features until gradients for local features vanish, while masked reconstruction keeps heads diverse. Revealing the Dark Secrets of Masked Image Modeling measures the same effect empirically: supervised models' deep-layer heads collapse, SimMIM's stay diverse. Both study pretraining. Neither asks what a 1%-label fine-tuning run does to the diversity that pretraining produced. That downstream question is ours. The theory's shortcut mechanism predicts that a small, strongly discriminative label signal should behave like the contrastive case.
Two papers address collapse during fine-tuning or in the representation, and they shape the design. Better Fine-Tuning by Reducing Representational Collapse shows that small-data fine-tuning discards pretrained features, and it regularizes output distributions under input noise (R3F). It never looks inside the attention layers, so it is a baseline rather than a mechanism we reuse. ContraNorm shows that token features can lose effective rank even when attention maps look diverse. This finding constrains us: head-map similarity alone is an insufficient diagnostic, so we also track the effective rank of head outputs and treat the two as separate quantities.
Several papers are the closest competitors on the method side. Layer Grafted Pre-training raises 1% ImageNet accuracy by stacking MIM-trained lower layers under contrastive upper layers. That is a pretraining fix; ours is a fine-tuning fix that could be applied on top of it. HoRA attacks head redundancy in low-data adaptation from the opposite direction, sharing parameters across heads through a hypernetwork, and its authors flag the risk of "over-sharing." Our constraint pushes against sharing at the level of attention activations, so comparing the two tests a real tension in the file: whether few-label adaptation benefits from heads pooling strength or from heads staying distinct. Absolute diversity losses, such as the cosine attention-diversification penalty in Robustifying Token Attention for Vision Transformers and the diversity loss of Diversified Visual Attention Networks, push all heads apart toward a fixed target. Ours only forbids moving closer than the pretrained model already was.
Research Program
The first stage is diagnostic. We fine-tune public ViT-B/16 checkpoints from MAE, MoCo v3, and Layer Grafted Pre-training, the last using the released code. We run each at 1%, 10%, and 100% of ImageNet-1K labels and log three quantities per layer throughout training. The first is head similarity, the mean pairwise similarity between heads' attention maps averaged over queries and a fixed probe set of images, in the style of the cross-head KL used by Xie et al. The second is the effective rank of concatenated head outputs. The third is functional redundancy, the accuracy drop when each head is ablated.
The prediction is that head similarity rises during fine-tuning and rises more at 1% than at 100%. It should also rise most from MAE's highly diverse starting point. Across seeds, the size of the rise should correlate with lower test accuracy. If similarity does not rise, or rises without predicting accuracy, the hypothesis is wrong and we report that. A clean negative result would itself tell practitioners that head diversity is not what they should protect.
The second stage is the intervention. Let $s^{ij}_l(\theta)$ be the similarity between heads $i$ and $j$ in layer $l$ under parameters $\theta$, with $\theta_0$ the pretrained parameters. The fine-tuning loss is
$$ \mathcal{L}(\theta) = \mathcal{L}_{\text{task}}(\theta) + \lambda \sum_{l}\sum_{i<j} \max\!\big(0,\; s^{ij}_l(\theta) - s^{ij}_l(\theta_0) - \delta\big), $$
where $\lambda$ weights the penalty and $\delta$ is a small tolerance. Heads may rotate, sharpen, or relocate their attention at no cost. Only merging is penalized. On a 12-layer, 12-head model the extra cost is 66 similarities per layer computed on maps the forward pass already produces. We also run a variant that applies the same hinge to the effective rank of head outputs, to test whether the ContraNorm distinction matters for adaptation.
The third stage transfers the question to a domain where labels are expensive and findings are spatially sparse. We fine-tune a chest-radiograph transformer at 1% and 10% labels and evaluate on CheXpert, NIH ChestX-ray, and SIIM-ACR pneumothorax segmentation. The backbone is the MRM or MGCA checkpoint if one is released; otherwise we pretrain MAE on MIMIC-CXR. We break results down by class frequency to test whether gains fall disproportionately on rare findings. Self-supervised representation learning from 12-lead ECG data found that minority classes are where pretraining helps most, so they are also where losing its benefit during fine-tuning should hurt most.
Baselines
The naive reference is standard full fine-tuning with the usual recipe, including layer-wise learning-rate decay, at the same label fraction. Linear probing is the zero-drift extreme. Every method gets the same labeled subsets (three fixed 1% splits), the same epochs, and the same hyperparameter search budget of 16 configurations each, selected on a held-out labeled validation split of matching size. Reporting across all three splits addresses the seed variance documented by Aghajanyan et al.
The strongest existing number on 1% ImageNet-1K with ViT-B/16 is 65.5% top-1 for Layer Grafted Pre-training, against 63.4% for MoCo v3 and 51.1% for MAE (Layer Grafted Pre-training). Because our method is a fine-tuning procedure, the fair test is not our method against grafting. It is grafting plus our constraint against grafting plus its own fine-tuning recipe, re-run under our protocol. We count a gain of at least one point, consistent across all three splits, as beating it. On CheXpert at 1% labels, the file reports 88.8% AUROC for MGCA (Multi-Granularity Cross-modal Alignment) and 88.5% mean AUC for MRM (Masked Record Modeling). On SIIM-ACR at 1% labels it reports a Dice of 71.9% for ASG (Anatomical Structure-Guided Pre-training). These are reference points under their own protocols. Our claim is judged against the same backbone fine-tuned without the constraint.
Four regularizer baselines compete directly. R3F (Better Fine-Tuning by Reducing Representational Collapse) is the strongest competitor that does not touch the attention layers. HoRA (HoRA) and vanilla LoRA are the parameter-efficient competitors, run on VTAB-1K, where HoRA reports 74.4% average accuracy versus 72.2% for LoRA. An absolute cosine diversity loss following Robustifying Token Attention tests whether the relative target matters.
The ablation that isolates our key ingredient is attention anchoring: a penalty on each head's divergence from its own pretrained attention map, tuned with the same budget. Anchoring preserves heads; our constraint only prevents merging. If anchoring does as well, the ingredient is preservation, not diversity.
Why not just…?
Why not just use a lower learning rate, a linear probe, or layer-wise decay so heads never move? These methods keep head diversity by keeping everything fixed. The failure of MAE at 1% labels suggests its features need real adaptation to become class-separable, so freezing trades one loss for another. Linear probing and tuned decay are both in our baseline set. If either matches the constraint at 1% labels, the lesson is that any drift is harmful and diversity is incidental, and we will report it that way.
Why not just anchor each head to its pretrained attention? This is the most serious threat, because anchoring also prevents merging, as a side effect. The difference is testable. Anchoring should hurt when the task needs heads to relocate, as when a head that tracked natural-image texture must learn to find a pneumothorax edge. The no-merge constraint allows that relocation. We compare the two directly, and we also measure how far each head's map moves under our method. If the heads that move most are also those whose ablation hurts downstream accuracy most, relocation is doing useful work and anchoring would have blocked it. If anchoring ties or wins, we drop the diversity framing.
Why not just pretrain with MIM, or with Layer Grafted Pre-training, and fine-tune normally? MIM gives diversity at pretraining time, yet it scores only 51.1% on 1% ImageNet versus 65.5% for grafting. Our diagnostic will show whether part of that gap comes from MAE's diversity being erased during fine-tuning, which would be the first evidence tying MAE's weak few-label performance to adaptation rather than pretraining. If MAE's heads stay diverse through fine-tuning and it still underperforms, the gap is purely semantic, and pretraining fixes like grafting are the right lever. Our method would then offer little on top.
Why not just use HoRA's cross-head sharing? HoRA and our constraint pull in opposite directions, so the comparison is informative whichever way it comes out. Running both alone and combined on VTAB-1K and at 1% labels shows whether sharing parameters and separating activations are complementary or in conflict.
Decisive Checks
The program succeeds only if three things hold. First, head similarity must measurably rise during 1%-label fine-tuning, and more than at 100%. Second, the no-merge constraint must beat attention anchoring and R3F at equal tuning budget on 1% ImageNet across all three splits. Third, the gains must be larger on rare CheXpert findings and SIIM segmentation than on common findings. If the first fails, we have a negative diagnostic result. If the second fails with anchoring as the winner, the finding is that preservation, not diversity, protects few-label adaptation, and we report it as such.
References
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers
- Revealing the Dark Secrets of Masked Image Modeling
- Better Fine-Tuning by Reducing Representational Collapse
- ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond
- Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations
- HoRA: Cross-Head Low-Rank Adaptation with Joint Hypernetworks
- Robustifying Token Attention for Vision Transformers
- Diversified Visual Attention Networks for Fine-Grained Object Classification
- Self-supervised representation learning from 12-lead ECG data
- Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning
- Advancing Radiograph Representation Learning with Masked Record Modeling
- Anatomical Structure-Guided Medical Vision-Language Pre-training