When a medical image classifier is trained on fewer than a hundred patients, the worry is less "what pixels does it look at" and more "which patients is it really copying from." A model that has learned the disease should base its call on a new chest X-ray mostly on earlier training cases that show the same pathology. A model that has learned a shortcut, such as a hospital's scanner contrast, a text marker, or the patient's sex, will instead base the call mostly on training cases from the same hospital or of the same sex. We propose to measure this directly. With so few patients, we can afford to retrain the model with each patient left out and record how every prediction changes. That record is a complete map of which training patients each decision depends on. In this setting, the small sample size is what makes the model's reasoning auditable.
The idea, in two sentences: build the patient-by-patient leave-one-out influence matrix of a small clinical model, which is affordable only because the cohort is tiny, and decompose it to find groupings of training cases that drive predictions but are not explained by the diagnostic label. Those groupings are candidate shortcuts, found without pixel masks, without a balanced reference model, and potentially without knowing in advance what the shortcut is. We then test whether this audit predicts the external-validation collapse that small clinical models are known to suffer, and whether down-weighting the offending patients prevents it.
Core Claim
Pixel-space explanations are the default tool for catching shortcuts, and they have a specific blind spot in small clinical data. Unveiling COVID-19 from Chest X-ray with deep learning showed that ResNets trained on about 447 multi-source chest X-rays sorted images by source institution rather than pathology. Cropping to the lungs with a U-Net removed text and hardware artifacts, but performance stayed modest (sensitivity 0.54, specificity 0.80 on CORDA), and the authors pointed to shortcuts inside the lung fields that cropping cannot reach. OSCAR found that biological sex works as an opaque shortcut inside anatomy on CheXpert and ADNI. A shortcut spread across intensity statistics or anatomy has no clean location, so a heatmap is a weak instrument for finding it. Such a shortcut does leave a trace in the training data: the predictions that rely on it depend on the training patients who share it. Our claim is that training-data attribution, measured exactly at small N, detects such diffuse shortcuts more reliably than pixel attribution or embedding similarity. We also claim that the audit score, computed only on internal data, ranks models by how badly they will fail on an unseen hospital.
The intersection here is real but partly occupied, so the contrast matters. EGxFSL uses Grad-CAM alignment with radiologist regions of interest inside a prototypical few-shot network and reports large gains. It needs dense expert masks, and it can only penalize shortcuts that sit outside the marked region. OSCAR localizes shortcuts by correlating attribution maps of the test model with those of a sensitive-attribute model and a balanced baseline model. Both reference models require labeled shortcut attributes and enough balanced data, which a cohort of 50 patients rarely has. Logit Attribution Matching enforces invariance of class-logit attributions across augmentation pairs. It requires knowing the nuisance in advance well enough to augment it away. HyDRA supplies the data-attribution machinery. It approximates leave-one-out influence by unrolling training, reaching a Spearman correlation of 0.986 with true leave-one-out influence against 0.820 for influence functions, but it was evaluated only on MNIST, Fashion-MNIST, and CIFAR-10 with random label noise and long training schedules. None of these works uses sample-level attribution to discover shortcuts, and none exploits the fact that at N below 100, exact retraining replaces approximation.
Research Program
The core object is an influence matrix. Let $f_{\theta}$ be the classifier trained on all $N$ training patients and $f_{\theta_{-i}}$ the same pipeline retrained with patient $i$ removed. For each held-out internal validation case $j$, we define
$$I_{ij} = \ell\big(f_{\theta_{-i}}(x_j), y_j\big) - \ell\big(f_{\theta}(x_j), y_j\big),$$
where $\ell$ is the cross-entropy loss and $y_j$ the label. A positive $I_{ij}$ means patient $i$ helped the model get case $j$ right. To make retraining cheap and stable, we freeze a pretrained backbone (RSNA-pneumonia or ImageNet weights) and fit the last block plus a linear head. With N ≤ 150 this takes minutes per fit. We average $I_{ij}$ over five seeds so that training noise is not mistaken for influence, and we estimate the noise floor from seed-to-seed variance.
The audit comes in two forms. The supervised form applies when acquisition metadata such as hospital or scanner exists, which in clinical data it often does. It compares how much of each case's positive influence comes from training patients who share its metadata group against how much comes from patients who share its label. These two quantities are confounded within the majority cells, so we compute them on the minority cells, for example positive cases from the hospital that contributed mostly negatives. The unsupervised form needs no metadata. We take the leading factors of a low-rank decomposition of $I$, regress out the component explained by the label-agreement structure, and inspect the residual factors. Each residual factor is a set of training patients who jointly drive predictions for reasons other than diagnosis. We then check post hoc whether these sets line up with site, sex, scanner, or view. If shortcuts exist, this is where they should show up.
Because full retraining gets expensive as N grows toward a few hundred, the second component tests HyDRA as a cheap stand-in in the short-schedule fine-tuning regime. Its error bounds assume a decaying learning rate over many epochs, and that assumption may fail here. Exact leave-one-out at small N provides the ground truth for this validation, which the original paper had only on toy benchmarks.
The third component is a remedy. Training patients who load heavily on a non-label factor are down-weighted in proportion to that loading, and the model is refit. This intervention uses only training data and the audit. It needs no masks, crops, or attribute labels.
Baselines
The naive reference point is plain fine-tuning (empirical risk minimization) of the same frozen-backbone model with standard augmentation. It shows how much shortcut damage there is to fix and how its internal accuracy overstates its external accuracy. The CORDA paper documents this failure mode, but the file gives only the in-cohort sensitivity of 0.54 and specificity of 0.80 and says that cross-dataset performance "plummeted." No reported cross-dataset number was found on Lacuna for CORDA, so we will measure it ourselves.
The strongest simple competitor, whenever site labels exist, is site-balanced reweighting: weight each hospital-by-label cell equally during training. Because it uses the true site labels, it serves as an oracle for the remedy. We run it on identical splits, with the same backbone and the same hyperparameter search budget of 20 configurations chosen on internal validation. Our unsupervised remedy beats it if it matches its external accuracy without site labels, or exceeds it when the dominant shortcut is not site (sex in ADNI, for example).
Lung cropping with a U-Net, following the CORDA paper, is the standard heuristic fix. We apply it both alone and combined with every other method.
The strongest existing auditor is OSCAR. On CelebA it raised ResNet-50 worst-group accuracy from 20.79% to 74.8% through test-time attenuation of shortcut regions. We give OSCAR its required sensitive-attribute model and its balanced baseline, trained on the full public dataset. That makes it an oracle with more data than our method sees. Matching its detection on parenchymal shortcuts while using only the small cohort would count as a win.
Where expert boxes exist, as in VinDr-CXR, we run EGxFSL's Grad-CAM alignment loss at its reported best weight of α = 0.10. It reported 76.12% accuracy against 56.21% for a plain few-shot baseline on VinDr-CXR. Our method is not expected to beat a mask-supervised method on in-distribution accuracy. The question is whether EGxFSL leaves intra-region shortcuts that our audit catches. Its accuracy on a deliberately confounded split decides this.
Two ablations isolate the key ingredient. The first swaps the influence matrix for embedding-space nearest-neighbor similarity between validation and training cases. The second swaps it for Grad-CAM++-based audits (Grad-CAM++). Everything downstream stays identical: the same decomposition and the same reweighting.
Why not just…?
Why not just balance the training set by hospital? If the only shortcut is a known site variable, this probably works about as well as our remedy, and we will report it as an oracle rather than argue otherwise. It fails in two cases the file documents. The first is a shortcut that is not the recorded site variable, such as sex in OSCAR's ADNI and CheXpert results. The second is a shortcut that is effectively unrecorded, such as acquisition settings inside the lungs. With at most ten positives per cell, balancing also throws away most of the minority data. The ADNI sex-confound experiment and the unrecorded-scanner experiment below are designed to settle whether it is enough.
Why not just crop to the organ? The CORDA study already did this and still saw weak performance, which its authors attributed to shortcuts inside the lung fields. OSCAR's sex results are likewise anatomical. Cropping is cheap and we include it everywhere, but it only addresses shortcuts outside the organ boundary.
Why not just cluster embeddings, as the t-SNE analysis in the CORDA paper did? This is the most serious threat. If embedding neighbors identify the same patient groups, the expensive retraining is unnecessary. Our argument is that embeddings record what the model represents, while influence records what its decisions depend on. A backbone can encode scanner identity strongly without using it. The embedding ablation tests this directly. If it matches the influence audit in both detection and its correlation with external failure, we will report that embedding similarity suffices and that the expensive audit adds nothing at this scale.
Why not just inspect Grad-CAM maps or run OSCAR? Grad-CAM-style maps are fragile. Robust Attribution Regularization reports top-k intersection under attack of only 8.12% for natural models on Oxford Flowers. Perturbation-based attributions also leave the data manifold (On Measuring Intrinsic Causal Attributions), and diffuse intensity shortcuts have no location to highlight. OSCAR fixes much of this but needs the reference models that small cohorts lack. We include both as baselines rather than dismissing them.
Proposed Experiments
The controlled experiment uses VinDr-CXR, which comes from two hospitals. We subsample training sets of N ∈ {36, 72, 144} in which hospital and label correlate at strength ρ ∈ {0, 0.5, 0.8, 0.95}, and evaluate on a test set where the correlation is reversed. Here the true shortcut is known. Detection is scored as the AUROC with which each audit's shortcut factor separates the two hospitals' patients. We then repeat with an unrecorded shortcut. A contrast or gamma shift applied to the positives of one group plants a known intensity confound inside the lungs, which heatmaps should struggle to localize. On ADNI we confound Alzheimer's labels with sex at small N, following OSCAR's setting. No raw ADNI performance numbers were found on Lacuna, so all numbers will be produced here. As a natural test, we train on public COVID-ChestXRay and evaluate on CORDA, and also train on CORDA and evaluate on SIIM-FISABIO-RSNA, reproducing the cross-dataset collapse reported in the CORDA paper.
Three decisive checks determine whether the idea holds. The first is predictive. Across roughly 60 trained models varying N, ρ, backbone depth (ResNet-18 against ResNet-50, which the CORDA study found overfits more), and cropping, the internal-only audit score should correlate negatively with external accuracy. We require Spearman ρ below −0.6 and a clear margin over the embedding and Grad-CAM++ audits. If the embedding audit does as well, the core claim fails. The second is the approximation check. HyDRA's rank correlation with exact leave-one-out should stay near its reported 0.986 under five-epoch fine-tuning; if it drops sharply, we will report that short-schedule training breaks its assumptions and restrict scalable use accordingly. The third is the remedy check. Influence-guided reweighting should close at least half the external-accuracy gap between plain fine-tuning and oracle site balancing on VinDr-CXR, and should beat oracle site balancing on the sex and planted-intensity confounds, at an equal tuning budget. Worst-group accuracy and expected calibration error will be reported alongside accuracy and AUROC. A failure of the third check alongside a pass of the first would still leave a useful result: an audit that tells a clinical team, before any external data arrives, that their model will not travel.
References
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Localising Shortcut Learning in Pixel Space via Ordinal Scoring Correlations for Attribution Representations (OSCAR)
- Expert-Guided Explainable Few-Shot Learning with Active Sample Selection for Medical Image Analysis
- Consistency Regularization for Domain Generalization with Logit Attribution Matching
- HyDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks
- Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks
- Robust Attribution Regularization.
- On Measuring Intrinsic Causal Attributions in Deep Neural Networks