Vision transformers (image models that split a picture into patches, or "tokens," and let every patch exchange information with every other) are expensive. Their cost grows with the square of the number of patches. On small medical datasets they also tend to memorize the wrong thing, such as a hospital's text overlay or a pediatric shoulder bone, instead of the disease. Two separate literatures attack these problems. Edge-efficiency work skips computation on "unimportant" patches. Interpretability work forces the model to rest its prediction on human-meaningful regions. This note argues that they can be one mechanism. If a model may only compute on a small, fixed number of patches per layer, and its final prediction may only read from those patches, then the set of patches it chose is a faithful explanation by construction. If that selection is trained to be grounded in the right anatomy, the tight budget becomes a bottleneck that a shortcut cannot squeeze through.
The proposal is to train this selection without the large teacher network that current token-skipping methods depend on, since small clinical cohorts cannot provide one. The selection is instead supervised by two cheap signals. The first is an invariance test: the prediction must not change when the patches the model skipped are scrambled. The second is a handful of expert region masks. Neither signal requires the second-order gradients that make existing attribution regularizers impractical. The central question is empirical and currently open. Under scarce, biased data, does shrinking the compute budget strip away shortcuts, or does it preferentially keep them because they are the most predictive tokens?
Core Claim
We claim three things about a vision transformer that keeps a fixed number $K$ of "selected" tokens per layer and routes the rest past the layer unchanged, so they can be reactivated later. First, reading the class decision only from the final selected set makes the selection a faithful attribution with zero inference overhead. Second, a first-order invariance loss on the skipped tokens plus a few expert masks is enough to ground that selection when training data number in the hundreds. Third, under this grounding, a smaller budget $K$ improves accuracy on held-out hospitals rather than hurting it. The same budget without grounding should do the opposite and retain shortcut tokens. If the third claim fails, that is also a useful result: it would mean edge compression and shortcut removal are in tension and must be designed separately.
Position Relative to Prior Work
The mechanism for reversible skipping comes from No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling. IdleViT splits tokens per layer into a computed set and an "idle" set that bypasses attention and feed-forward blocks and can return later. It cuts GMACs by 33% on DeiT-S at 79.6% versus 79.8% ImageNet top-1. Its partitioning, however, is trained with a graph-cut loss and 30 epochs of distillation from a full-size teacher. We keep the idling mechanism and replace both the graph-cut loss and the teacher. Other compression work that drops tokens permanently, namely Patch Slimming for Efficient Vision Transformers and the whole-slide pruner in Scalable Whole-Slide Vision-Language Modeling with Learned Token Pruning, motivates our choice of reversibility. SLIM reports that kept tokens land on diagnostic tissue, but aggressive pruning degraded performance on EBRAINS, and it relies on 7,568 slides and a frozen pathology encoder. OTPrune selects tokens training-free by matching the full token distribution. Its implicit goal is to preserve everything, including shortcuts, which is exactly the opposite of what a biased small dataset needs.
The training signal borrows from attribution regularizers while avoiding their cost. Improving Deep Learning Interpretability by Saliency Guided Training masks the least-salient inputs and penalizes any change in output. It needs input gradients to find those inputs. In our model the idle set is the low-saliency set, so the same penalty costs one extra forward pass and no differentiation through gradients. Fourier-transform-based attribution priors supplies the motivating evidence that an attribution prior can substitute for data: their model matched full-data performance with 1% of the data. Their prior is second-order and was shown only on 1D CNNs.
Three papers set the bar for the small-sample side, each from a different angle. Mask-guided Vision Transformer (MG-ViT) for Few-Shot Learning is closest in spirit. It zeroes out background patches using Grad-CAM, which lifts Agri-ImageNet 1-shot accuracy from 74.9% to 85.6%. It keeps all $N$ tokens, so it saves no compute, and it needs a gradient pass to make its masks. Unlearning Spurious Correlations in Chest X-ray Classification (eXBL) is the main constraint on our claims. Pushing a CNN off a growth-plate shortcut with only two exemplar explanations raised lung activation precision from 0.32 to 0.35 but cut in-source accuracy from 0.95 to 0.90. We therefore do not promise in-source gains, and we make held-out-source accuracy the primary metric. Unveiling COVID-19 from Chest X-ray with deep learning shows why: transferred models clustered by source hospital and reached only 0.54 sensitivity and 0.80 specificity on CORDA.
Two interpretability-by-design transformers show the opposite tendency from ours. Interpretability-Aware Vision Transformer attaches an interpreter head that adds compute. DOT-CBM grounds concepts to patches by optimal transport and gains 3.29% on CUB, but it uses large data and full token sets. Our version removes computation instead of adding it.
Research Program
The model is an ImageNet-pretrained DeiT-S, a deliberately general-domain initialization because no domain teacher exists. A one-layer linear scorer is inserted before chosen blocks. At each such block it keeps exactly $K_\ell$ tokens via a straight-through top-$K$, and the rest idle forward as in IdleViT. Keeping $K_\ell$ fixed rather than thresholded gives static tensor shapes. This addresses the known gap between FLOP savings and real latency on edge accelerators, which handle variable-length sequences poorly. The class prediction $f(x)$ is pooled only from the tokens selected at the last block. The training objective on a batch is
$$ \mathcal{L} = \mathrm{CE}\big(f(x), y\big) + \lambda\, \mathrm{KL}\big(f(x)\,\|\, f(\tilde{x})\big) + \mu\, \mathbb{1}[m \text{ given}]\, \mathrm{BCE}\big(s(x), m\big), $$
Here $\tilde{x}$ is the input with the patches idle in the final selected set replaced by patches shuffled from other images, $s(x)$ is the average selection score per patch, $m$ is a binary expert region mask available for only a few images, and $\lambda, \mu$ are weights. The invariance term makes skipped patches irrelevant to the prediction. The mask term tells the scorer which irrelevance is correct. No teacher appears anywhere.
The analysis has three parts. First, we trace accuracy on held-out sources against the budget $K$, with and without grounding. This directly tests whether compression removes or preserves shortcuts, a question the literature leaves open. Second, we measure how many expert masks are needed, sweeping from zero (invariance only) to 2, 10, and 50 masks. The 2-mask point mirrors eXBL's exemplar budget. Third, we check faithfulness: deletion and insertion curves on the selected set, and whether selections stay stable across random seeds, using the Jaccard-overlap measure of Tseng et al.
Proposed Experiments
The primary testbed is the six-source, four-class Covid-19 CXR dataset used by eXBL (5,200 images, 800 per class for training). Its known shortcut is the pediatric growth plate in the viral-pneumonia images. We train on subsets of 25, 100, and 800 images per class drawn from a subset of sources. We test on in-source data, on held-out sources, and, for the COVID-versus-other task, on CORDA's 447 radiographs. Lung masks supply both $m$ and the activation-precision metric. A controlled variant plants a label-correlated text tag in training images and removes the correlation at test time. This gives a known ground truth for whether the selection idles the shortcut.
Two secondary benchmarks test generality beyond radiography. On CUB-200-2011 with 5 and 10 images per class, part annotations allow part-localization scoring of the selected set. On Agri-ImageNet 1-shot and 5-shot, we compare directly with MG-ViT's reported numbers. A full-data ImageNet-1K run serves only as a sanity check that the teacher-free objective does not break efficiency. The comparison point there is IdleViT's 79.6% at 33% fewer GMACs. Compute is reported as GMACs and as measured batch-1 latency on an ARM CPU and an edge NPU, using static shapes.
Baselines
The naive reference is full fine-tuning of DeiT-S on each subset with weight decay and standard augmentation, using all tokens. On Agri-ImageNet this "Ft-full" setting is reported at 74.9% 1-shot (MG-ViT). On the CXR dataset we found no reported result on Lacuna for a vision transformer, so we will establish this baseline ourselves.
The strongest competitor on the few-shot accuracy question is MG-ViT. It is the one method that already combines a ViT, interpretability cues, and few-shot data, with 85.6% 1-shot and 98.1% 5-shot on Agri-ImageNet. We rerun it on our splits from the same initialization, with the same augmentation and the same tuning budget of 20 trials per method. Beating it means at least matching its accuracy at no more than half its GMACs, with no gradient pass at inference.
The strongest competitor on shortcut removal is eXBL. It reports 0.90 in-source accuracy and 0.35 lung activation precision against an unrefined model's 0.95 and 0.32. We run it with its CNN backbone and also ported to DeiT-S, using the same two exemplars. Beating it means higher held-out-source accuracy and activation precision at equal annotation cost.
The strongest efficiency competitor is IdleViT with its original graph-cut loss and distillation. Its teacher is a full DeiT-S fine-tuned on the same small subset, because that is the only teacher available in this regime. It runs at identical $K_\ell$ and epochs. Beating it means equal or better held-out accuracy without a teacher.
We also include EViT-style attention-magnitude pruning and OTPrune applied training-free to the fine-tuned full model at the same budget. Finally, three ablations isolate the key ingredients. The first lets the classifier read all tokens, which tests whether the bottleneck matters. The second uses irreversible dropping, which tests whether reactivation matters. The third removes the invariance term, which tests whether masks alone suffice.
Why not just…?
Why not just take MG-ViT's Grad-CAM masks and drop the masked tokens instead of zeroing them? This is the most serious threat, because it would deliver compute savings with an existing method. It has three weaknesses. It needs a backward pass per image at inference to make the mask. Its mask comes from a model that has already learned whatever shortcut the data contains. And dropping tokens is irreversible, which SLIM's EBRAINS degradation suggests is fragile. We run exactly this "MG-ViT plus hard drop" variant at matched $K$. If it equals our held-out accuracy on the planted-tag and CXR experiments, our contribution narrows to removing the inference gradient pass, and we will say so.
Why not just deploy a small CNN with eXBL? A ResNet-18 avoids the overfitting Tartaglione et al. saw with ResNet-50 and runs cheaply. But eXBL's own results show the CNN prior does not prevent the growth-plate shortcut: the unrefined model reached 0.95 accuracy at 0.32 lung precision. eXBL also relies on Grad-CAM in the training loop. It remains a baseline at matched latency, so if it wins on held-out sources the ViT is not worth its cost.
Why not just freeze a pretrained model and prune tokens training-free? This costs no training and suits tiny data. However, Tartaglione et al. found transferred features cluster by hospital source, with CORDA sensitivity of 0.54. OTPrune's goal of preserving the full token distribution would retain those source cues. The training-free OTPrune and EViT baselines test this directly on held-out sources.
Why not just distill IdleViT from a teacher fine-tuned on the small set? That teacher is precisely the overfit, shortcut-bearing model the data produces, and distillation would copy its preferences into the selection. The teacher-distilled IdleViT baseline, scored on the planted-tag test, measures whether this happens.
Decisive Checks
The idea stands or falls on two results. The first is the shape of the curve relating held-out-source accuracy to budget $K$ on CXR and the planted-tag variant. Grounded idling should rise or hold as $K$ shrinks. Ungrounded idling and teacher-distilled IdleViT should fall, and their selected sets should contain the tag. The second is a seed-stability and deletion test showing that the selected set is a more faithful and more stable explanation than Grad-CAM on the same model, at no inference cost. If grounded and ungrounded curves coincide, then the budget alone neither helps nor hurts shortcuts, and the interpretability claim reduces to "a cheap saliency map." If grounding works only with 50 or more masks, the method is an annotation-hungry efficiency tool rather than a small-sample one. Either outcome is reportable, and both are decided by experiments already in the plan.
References
- No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
- Patch Slimming for Efficient Vision Transformers
- Scalable Whole-Slide Vision-Language Modeling with Learned Token Pruning
- OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport
- Improving Deep Learning Interpretability by Saliency Guided Training
- Fourier-transform-based attribution priors improve the interpretability and stability of deep learning models for genomics
- Mask-guided Vision Transformer (MG-ViT) for Few-Shot Learning
- Unlearning Spurious Correlations in Chest X-Ray Classification
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Interpretability-Aware Vision Transformer
- Discovering Fine-Grained Visual-Concept Relations by Disentangled Optimal Transport Concept Bottleneck Models