Hospitals want small, fast image models that run on modest hardware, so large medical vision models are increasingly compressed into compact students by knowledge distillation. In knowledge distillation, a small "student" network is trained to imitate the outputs or internal features of a large "teacher." Medical image models are also known to cheat. They learn "shortcuts," such as scanner noise, hospital position markers, or skin-pen marks, which predict the label inside one hospital and fail at the next one. These two facts have so far been studied separately. Compression work reports accuracy averaged over many cohorts. Shortcut work uses convolutional networks and ignores cost. This proposal joins them. It asks whether a compressed vision transformer copies its teacher's shortcuts or makes them worse. It then tests whether a second, tiny "specialist" teacher can stop that, where the specialist is trained on a few percent of carefully cleaned images and supervises the student's intermediate layers. Every claim is judged only on hospitals the models never saw during training.
In two sentences: distill a large vision transformer into a small one as usual, but add a guard. Linear readouts at the student's intermediate layers are pulled toward those of a specialist trained on a small clean subset. The test is whether this keeps the efficiency gains of compression while closing the gap between internal and external accuracy that shortcut learning opens.
Core Claim
We claim that compressing a vision transformer under ordinary distillation amplifies shortcut reliance in a measurable way. The student has less capacity, so it favours the cheapest predictive cue, and in medical images that cue is often an acquisition artifact. We further claim that intermediate-layer distillation from a small clean-data specialist can remove most of this amplification at almost no cost at inference time. The method was introduced for convolutional networks by Preventing Shortcut Learning in Medical Image Analysis through Intermediate Layer Knowledge Distillation from Specialist Teachers. The result should hold under strict leave-hospital-out external testing.
Three groups of prior work shape the claim.
The first group motivates the gap. A Deployment-Friendly Foundational Framework for Efficient Computational Pathology (LitePath) distills three pathology foundation models into a 22.5M-parameter ViT-Small. It reports 99.71% of Virchow2's AUC at 403.5x fewer FLOPs, but it never asks whether the student relies on site artifacts. Rethinking Annotation Granularity for Overcoming Shortcuts in Deep Learning–based Radiograph Diagnosis: A Multicenter Study shows that image-level chest X-ray classifiers exploit hospital markers. It also shows that adding uncurated data such as CheXpert can lower performance on some diseases. Finally, Eye-Gaze-Guided Vision Transformer for Rectifying Shortcut Learning shows that vision transformers also attend to background artifacts. Its fix requires radiologist eye-tracking and was not validated externally.
The second group supplies the method. Boland et al. train a specialist on 5–20% clean data and attach linear probes to its intermediate layers. They then regularize a student trained on the full biased data by matching probe outputs layer by layer. This cut the true-positive-rate gap on a noise shortcut from 1.000 to 0.377 and raised external AUC on MIMIC from 0.461 (standard training) to 0.733. The authors explicitly leave vision transformers as open future work, and they report no compute analysis.
The third group constrains the design. External validation of predictive models for diagnosis, management and severity of pediatric appendicitis shows that retraining with 80% of the target hospital's data moved diagnosis AUROC only from 0.85 to 0.87, against 0.96 internally. Fixing things afterwards at the target site therefore cannot be assumed to work. Wang et al., as summarized in Independent External Validation for Clinical Predictive Models, warn that pooling sites before a random split overstates how well a model generalizes. Our evaluation therefore never pools a test hospital into training.
Research Program
The student is a ViT-Small. It has two teachers.
The first is a large pretrained vision transformer that has been fine-tuned on the full, biased source data. A DINOv2-class backbone is the natural choice, since Enhancing Compositional Generalization via Compositional Feature Alignment shows such models still fail under domain–class shifts. This teacher supplies capacity and soft labels, in the tradition of Transferring knowledge from a RNN to a DNN.
The second teacher is a specialist ViT trained only on a small cleaned fraction $\rho$ of the source data, with $\rho$ ranging from 5% to 20%. In the cleaned images, the shortcut is absent or decorrelated from the label. We attach linear probes to both specialist and student at matched depths. The student is trained with
$$ \mathcal{L} = \mathrm{CE}(y, p_S) + \alpha\,\mathrm{KL}\big(p_{F}\,\|\,p_S\big) + \lambda \sum_{l \in \mathcal{S}} \mathrm{KL}\big(q^{l}_{T}\,\|\,q^{l}_{S}\big). $$
The symbols are as follows. $y$ is the label. $p_S$ and $p_F$ are the final predictions of the student and the foundation teacher. $q^{l}_{T}$ and $q^{l}_{S}$ are the class predictions of the linear probes at layer $l$ of the specialist and the student. $\mathcal{S}$ is the set of probed layers. $\alpha$ and $\lambda$ weight the two distillation terms.
Three questions about transformers do not arise for convolutional networks, and the program answers each.
The first is what to probe. The options are the class token, mean-pooled patch tokens, or attention-weighted pooling. We expect pooling over patch tokens to matter, because shortcuts such as markers are spatially local.
The second is depth matching across different widths. The foundation teacher has twelve or more blocks and the student is shallower, so we map layers by relative depth.
The third is the interaction with token pruning. Efficient transformers often drop patches before or during inference. LitePath's Adaptive Patch Selector is one example; EG-ViT's gaze mask is another. We measure what fraction of kept tokens fall on known shortcut regions. That fraction tells us whether the efficiency mechanism is itself selecting for artifacts.
Alongside accuracy we report parameters, FLOPs, and measured latency on one data-center GPU and one edge device. Any robustness gain must be shown to come without giving up the efficiency gain.
Baselines
The naive reference point is a ViT-Small trained by ordinary empirical risk minimization, meaning cross-entropy on the full biased source data with no teacher. It tells us how much shortcut reliance a small transformer acquires on its own. The closest reported analogue is the convolutional ERM result in Boland et al.: 0.461 AUC on external MIMIC. No reported result was found on Lacuna for a ViT trained this way on the same benchmark.
The second baseline is standard compression with no guard. This is a ViT-Small distilled from the foundation teacher with the $\alpha$ term only, in the style of LitePath's feature and logit distillation. It is the strongest competitor on efficiency. LitePath reports 99.71% retention of Virchow2's AUC across 37 pathology cohorts, so a reviewer may reasonably expect it to be "good enough." Our method beats it only if three things hold together: external AUC is higher on hospitals never seen in training, the shortcut gap is smaller, and the inference cost is the same, which holds by construction because the probes are discarded after training.
The strongest existing anti-shortcut method is Boland et al.'s intermediate-layer specialist distillation, applied in its original convolutional form at matched parameter count. It reports a gap reduction from 1.000 to 0.377 and 0.733 external MIMIC AUC, against 0.737 for a clean baseline. We rerun it on our own splits rather than copying numbers, because our hospitals and shortcuts differ. A ViT version beating it means two things. First, comparable or better external AUC at equal or lower measured latency. Second, a shortcut gap no worse than the convolutional version.
Every method gets the same source images, the same clean subset, and the same augmentation. Each also gets the same budget of hyperparameter trials, selected on a held-out split from the source hospital only. Test hospitals are never used for selection.
Two ablations isolate the key ingredient. In the first, we keep the specialist but distill only its final output, which tests whether intermediate layers are what matter in transformers. In the second, we drop the specialist and keep only the foundation teacher. Comparing the second ablation with the naive baseline measures shortcut amplification under compression directly.
Why not just…?
Why not just train the small model on the clean subset alone? This is the real threat. In Boland et al. the clean baseline (0.737) slightly exceeds the distilled student (0.733) on MIMIC. If a ViT-Small trained only on 5–20% clean data matches our method externally, the specialist-distillation idea is unnecessary in that regime. We would report that and narrow the claim to where the clean subset is too small to train a transformer by itself. We sweep $\rho$ from 5% to 20%, expecting clean-only transformers, which are data-hungry, to fall off at small $\rho$ faster than distilled students. The proposal is only worth pursuing if such a crossover exists.
Why not just use augmentation such as MixUp? Boland et al. report that intermediate-layer distillation significantly outperformed MixUp on the CheXpert noise shortcut. That result is for convolutional networks, so we include MixUp under the same budget rather than assume it transfers.
Why not just fine-tune on the target hospital? Target labels are often unavailable before deployment. Where they were available, Marcinkevičs et al. found retraining recovered diagnosis AUROC only from 0.85 to 0.87. We include target fine-tuning as an upper-bound reference on one external site. It is not treated as a competitor.
Why not just rely on the big teacher's scale? CG-Bench results in Wang et al. show that CLIP and DINOv2 still struggle with unseen domain–class combinations, so scale alone does not remove shortcuts. Our foundation-only ablation tests whether its student does any better.
Proposed Experiments
For chest X-rays, we train on CheXpert with controlled shortcuts injected following Boland et al.: additive noise and visual markers correlated with the label. We also train on natural, uninjected data. External tests are MIMIC-CXR, PadChest, and NIH-Google, none of which are ever pooled into training. For the NIH ChestX-ray14 subset that has bounding boxes, we compute the overlap of attention and kept tokens with lesions, as a localization check on what the student looks at. For dermatology, we train on ISIC 2017, with its pen-mark shortcuts, and test externally on Fitzpatrick17k. SimBA's synthetic 3D brain MRI provides a fully controlled test of the shortcut gap.
Metrics are external AUC, the true-positive-rate gap between shortcut-present and shortcut-absent images, parameters, FLOPs, and edge latency. We report bootstrap confidence intervals over three seeds.
Three checks decide the outcome. The first tests amplification: the foundation-only distilled student shows a larger shortcut gap than its teacher. If it does not, the motivating claim fails and we report compression as shortcut-neutral. The second tests the remedy: specialist intermediate distillation reduces that gap and raises external AUC over standard compression at identical inference cost. The third tests the crossover: at $\rho = 5\%$ the guarded student beats a clean-only ViT-Small externally. A secondary analysis asks whether token pruning in the guarded student keeps fewer shortcut-region tokens than in the unguarded one. That would show the guard also makes the efficiency mechanism safer.
References
- Preventing Shortcut Learning in Medical Image Analysis through Intermediate Layer Knowledge Distillation from Specialist Teachers
- A Deployment-Friendly Foundational Framework for Efficient Computational Pathology
- Rethinking Annotation Granularity for Overcoming Shortcuts in Deep Learning–based Radiograph Diagnosis: A Multicenter Study
- Eye-Gaze-Guided Vision Transformer for Rectifying Shortcut Learning
- External validation of predictive models for diagnosis, management and severity of pediatric appendicitis
- Comparison of deep learning with traditional models to predict preventable acute care use and spending among heart failure patients
- Enhancing Compositional Generalization via Compositional Feature Alignment
- Transferring knowledge from a RNN to a DNN