When a large pretrained vision model is adapted to a medical task using only a handful of patients from one hospital, two failures tend to happen together. The adapted model loses its wide view of the image: each output comes to depend on a small neighborhood of pixels. It also fails when moved to another hospital's scanner. Recent work treats these as separate problems. One line restores the lost view with very large convolution kernels but tests only on held-out images from the same hospital. Another line makes models robust to new scanners but trains on the full source dataset. This note argues that the two failures are linked, and that the link is not simply how wide the model looks. What matters is where the widened view lands. A wide view that falls on the organ helps cross-site transfer. A wide view that falls on site-specific context such as image borders, burned-in text, patient positioning, or scanner texture actively hurts it.
We propose to test this directly and to build an adapter around it. The adapter is a small add-on module inserted into a frozen pretrained model. Ours applies several dilated convolutions in parallel, where each convolution spaces out its kernel weights to reach farther without adding parameters. It then mixes these scales separately at each pixel, using a selection map that is trained to stay the same when only the scanner "style" of the image changes. The measurable prediction is this: across methods, seeds and data sizes, out-of-site accuracy after few-patient adaptation should track the fraction of the model's pixel influence that falls outside the target anatomy much more closely than it tracks the overall size of that influence.
Core Claim
The effective receptive field (ERF) is the map of how strongly each input pixel influences a given output. It is measured by gradient magnitude. Dual-Kernel Adapter (DKA) showed that standard adapters and LoRA shrink the ERF when trained on under about 1% of the data, and that this shrinkage makes them worse than a linear probe. On COVID chest X-rays at 0.63% data, adapters reach 83.29% and LoRA 80.43%, against 86.84% for linear probing. DKA fixes the problem with parallel 51×51 and 5×5 depthwise kernels, reaching 89.01%.
We claim that under single-site, few-patient adaptation, the ERF size DKA restores is necessary but not sufficient for cross-site generalization. The shortcut literature shows why. In Shortcut Learning in Medical Image Segmentation, models used calipers, text and image-border statistics as proxies for anatomy. In Unveiling COVID-19 from Chest X-ray, features clustered by source hospital rather than by pathology. A fixed 51×51 kernel restores reach uniformly, and so it gives the model more access to exactly these cues. Our specific claim is that making the per-pixel choice of scale insensitive to scanner style keeps restored reach on anatomy. We also claim that this choice, not kernel size, separates adapters that transfer from adapters that do not.
The diagnostic we use throughout is the off-anatomy fraction. For an output location $p$ inside the target structure, let $E_p(q)$ be the normalized gradient magnitude $|\partial y_p / \partial x_q|$, summing to one over input pixels $q$. Let $A$ be the ground-truth anatomy mask, dilated by a fixed margin so that genuinely useful peri-organ context is not counted as off-anatomy. Then
$$\rho = \frac{1}{|A|}\sum_{p \in A} \sum_{q \notin A} E_p(q),$$
which is the average share of influence that each organ pixel draws from outside the organ's neighborhood.
Relation to Existing Work
Three papers motivate the gap. DKA (Dual-Kernel Adapter) diagnoses ERF collapse and restores it, but evaluates only in-domain splits and uses fixed kernel sizes. It flags adaptive kernel weighting only as preliminary future work. We keep its setting and its insertion point, and change what the added module computes. The shortcut studies (Shortcut Learning in Medical Image Segmentation, Unveiling COVID-19) show that the cues small medical datasets offer are often non-anatomical. They fix this with preprocessing: inpainting, reflection padding, or cropping to the lung field. We ask whether an architecture can avoid these cues without knowing them in advance. AMOS shows that MRI models lose over 20% Dice on unseen vendors. That result makes the target concrete.
Three papers constrain the design. Selective Sparse Sampling (S3N) warps images so that a second branch attends to low-response regions. It reports that this branch locks onto textured background. This is a warning that "look at more places" is dangerous without a filter. S3N also needs multiple backbone passes, which is why we choose a single-pass per-pixel gate instead. PSConv and CFPNet show that multi-scale dilation can be nearly free in parameters. Both, however, assign dilation rates statically regardless of image content, and PSConv reports a 1.06× latency cost from non-uniform dilation. We borrow their parallel-dilation and hierarchical-summing ideas to avoid gridding, the holes in coverage that a single large dilation leaves. We then replace the static assignment with a content-dependent one.
Three papers supply methods. FreeSDG and TinyMIG both rest on the observation that scanner differences live mostly in the Fourier amplitude (low-frequency "style") while anatomy lives in the phase. FreeSDG builds augmentation from this with Gaussian-filtered frequency mixing. TinyMIG builds phase-based alignment from it. We reuse the amplitude perturbation to create style-shifted views and the phase-only reconstruction to feed the scale gate. Unlike both papers, we work under few-patient budgets, and we apply the invariance to where and at what scale the adapter looks rather than to output features. MADGNet combines frequency filtering with multi-scale attention, but at 31M parameters, with full data, and without dilation. CollaSU-SDG pairs a specialist model with a frozen foundation model and adds low-rank adapters, but it runs both models and uses full data.
Research Program
The module is a style-invariant scale-gated dilated adapter. It sits in the same bottleneck position as DKA inside a frozen ViT encoder. It reshapes tokens into a 2D map $z$ and applies depthwise 3×3 convolutions at dilations $d_k \in \{1,2,4,8,16\}$. The largest rate reaches roughly the span of DKA's 51×51 kernel at patch resolution, at a small fraction of its parameters. Following CFPNet, the module sums branches hierarchically to suppress gridding, and it mixes them per pixel:
$$h(p) = \sum_k g_k(p)\,\mathrm{DW}_{d_k}(z)(p), \qquad g = \mathrm{softmax}\big(W * \phi(z)\big),$$
where $g_k(p)$ is the weight given to scale $k$ at location $p$, $W$ is a small convolution, and $\phi(z)$ is the phase-only reconstruction of $z$: the feature map with its Fourier amplitude replaced by a constant. This makes the gate's input largely blind to style.
Training adds one term on top of the task loss. For each source image $x$, we create an amplitude-perturbed view $\tilde{x}$ with FreeSDG-style frequency mixing. We penalize $\lVert g(x) - g(\tilde{x}) \rVert_1$, so that the choice of scale at each location cannot depend on scanner style. The task loss is applied to both views. We keep DKA's asynchronous learning rates for adapter and head so that the comparison isolates the module itself.
The analysis side matters as much as the module. For every trained model, we compute ERF size (area containing 90% of influence) and the off-anatomy fraction $\rho$ on source-site test images. We then relate both to out-of-site Dice. If $\rho$ explains cross-site variance that ERF size does not, the mechanism holds regardless of whether our module wins.
Baselines
The naive reference is linear probing on the frozen encoder. It is the method DKA found to beat standard adapters at under 1% data. Because it adds no spatial adaptation, it shows what the pretrained features alone give across sites. We also run standard bottleneck adapters and LoRA, since DKA reports that they collapse under scarcity and we want to see whether that collapse is worse out of site.
The strongest in-setting competitor is DKA itself, run from the released code misswayguy/DKA with the same frozen backbone, the same few-patient source subsets, and the same augmentation. It is the strongest competitor because it already solves ERF collapse, and our claim is precisely that solving collapse is not enough. DKA's reported in-domain numbers are 89.01% accuracy on COVID at 0.63% data and 23.02% mIoU on BRATS at 1.25%, where full fine-tuning gets 22.39%. No reported result found on Lacuna for DKA under cross-site or leave-one-domain-out evaluation. We will therefore establish that number ourselves.
The strongest single-source domain generalization methods run with full data are CollaSU-SDG, reported at 81.29% average Dice on six-site prostate MRI, TinyMIG, reported at 90.09% Dice on fundus, and FreeSDG, reported at 0.795 Dice on LES-AV and 0.736 on IOSTAR with DRIVE as source. All three will be retrained on our few-patient subsets with their published recipes. Their full-data figures serve only as ceilings, since no reported result found on Lacuna for any of them below full source data.
Every method gets the same backbone where applicable, the same subsets and seeds, and an equal hyperparameter search budget of 20 trials tuned on a held-out portion of the source site only. No target-site data is used for model selection.
Two ablations isolate the key ingredient. The first gives the gate raw features instead of phase-only features and drops the consistency term. The second replaces the per-pixel gate with PSConv-style static cyclic dilation. Beating DKA means higher mean out-of-site Dice across held-out sites at the 5- and 10-patient budgets, with confidence intervals over at least five subset draws that exclude zero difference, and in-domain performance no worse than DKA.
Why not just…?
Why not just combine DKA with frequency augmentation? This is the real threat. DKA restores reach, and FreeSDG's augmentation, which the FreeSDG ablation credits with raising LES-AV Dice from 0.720 to 0.795, removes style. The combination might be all that is needed. We run it as a named baseline with the identical augmentation our method uses. If it matches our module on out-of-site Dice and also shows a low off-anatomy fraction, then the gate is unnecessary, and the contribution reduces to the diagnostic and the finding that augmentation suffices. Our prediction to the contrary rests on the shortcut-injection test below. Amplitude augmentation does not remove a burned-in text tag or a border artifact, and a fixed 51×51 kernel sees them anyway.
Why not just linear-probe a foundation model, since those are already robust to new scanners? Linear probing preserves pretrained robustness but cannot reshape spatial features. DKA's BRATS results show adapted models ahead in dense prediction at 1.25% data. We will report linear probing out of site at every budget. If it matches adapted models on cross-site Dice for classification but trails them on segmentation, that locates where spatial adaptation earns its cost.
Why not just crop to the anatomy, as the COVID study did? For classification, cropping to the lung field is a legitimate and strong fix that Unveiling COVID-19 validated. We include it as a baseline on the chest X-ray task. For segmentation, cropping needs the answer it is supposed to produce. It also removes useful peri-organ context and does nothing about scanner texture inside the organ. Our module aims to get a comparable off-anatomy fraction without an external segmenter.
Proposed Experiments and Decisive Checks
The main testbed is six-site prostate MRI (NCI-ISBI13, I2CVB and PROMISE12), evaluated leave-one-domain-out. We add fundus optic disc and cup segmentation (REFUGE, RIM-ONE-r3, ORIGA, Drishti-GS) and vessel segmentation with DRIVE as source and LES-AV and IOSTAR as targets. In each, the source site is subsampled to 5, 10 and 20 patients or images, plus full data, with five draws per budget. For classification we use the CORDA and COVID-ChestXRay cross-dataset setting from eidoslab/unveiling-covid19-from-cxr. There we report sensitivity and specificity against the paper's ResNet reference of 0.54 sensitivity and 0.80 specificity on CORDA. Metrics are Dice or accuracy, both in-site and out-of-site, together with ERF size, the off-anatomy fraction $\rho$, and latency.
Three checks decide the claim. The first is correlational. Across all trained models, we regress out-of-site Dice on $\rho$ and on ERF size. The hypothesis fails if ERF size explains as much variance as $\rho$.
The second is interventional. We burn a synthetic site marker, a text tag or a bright border band, into the source-site images only, following the artifact types documented in the shortcut study. We then measure the out-of-site drop for each method on clean targets. Our prediction is that DKA's drop exceeds linear probing's, because of its wider fixed reach, and that our module's drop is smallest. If DKA plus augmentation resists the marker as well as our module does, we concede the gate.
The third is the gate ablation. If removing phase-only input and the consistency term leaves out-of-site Dice and $\rho$ unchanged, then per-pixel scale selection is not doing the work we attribute to it.
A positive result on all three would mean that small-data medical adapters should be judged and designed by where their influence falls, not by how far it reaches.
References
- Dual-Kernel Adapter: Expanding Spatial Horizons for Data-Constrained Medical Image Analysis
- Shortcut Learning in Medical Image Segmentation
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Selective Sparse Sampling for Fine-Grained Image Recognition
- PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer
- CFPNET: Channel-Wise Feature Pyramid For Real-Time Semantic Segmentation
- Frequency-Mixed Single-Source Domain Generalization for Medical Image Segmentation
- TinyMIG: Transferring Generalization from Vision Foundation Models to Single-Domain Medical Imaging
- Modality-Agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attention
- Perturbating, Tuning, and Collaborating: Harnessing Vision Foundation Models for Single Domain Generalization on Medical Imaging