Which Attention Axis to Retrain: Shift-Guided Adaptation of Block/Grid Transformers for Tiny Clinical Cohorts

Many vision transformers now split attention into two complementary operations. One attends densely inside small local windows, which captures texture and edges. The other attends sparsely across a coarse grid that spans the whole image, which captures layout. When such a model, pretrained on a large dataset, must be adapted to a clinical task with only one to twenty labeled scans, that split gives a natural way to choose what to retrain. Our claim is that the two pathways carry different kinds of transferable knowledge. When a new site or modality changes how tissue looks but not where anatomy is (contrast versus non-contrast CT, or CT versus MRI), only the local pathway should be retrained and the global grid pathway should be frozen. When the new task changes what structure is sought while the appearance stays the same, the global pathway has to move. We further propose that the choice can be made without labels, by measuring on unlabeled target scans which pathway's internal features shift most from the source distribution.

This matters because the regime is real and unforgiving. Specialized clinical models are often restricted to private cohorts of fewer than 100, and sometimes fewer than 20, volumes (Overcoming Data Scarcity in Specialized Medical Deep Learning). At that size, every extra trainable parameter is an opportunity to memorize scanner artifacts. Low-rank adaptation (LoRA, which adds small trainable low-rank matrices beside frozen weights) is the default answer, but it is usually spread uniformly across layers. A model whose computation is already factored into "local appearance" and "global layout" operators offers a principled, testable alternative to uniform spreading.

Core Claim

In a 3D multi-axis transformer, putting the adaptation budget on the pathway whose features shift under the target distribution will beat spending the same budget uniformly or by depth. The specific predictions are these. Under appearance shift with preserved layout, adapting only the convolution and block pathways will match or exceed uniform LoRA at equal trainable-parameter count, and will be more stable across random draws of the support scans. Under layout or target-structure shift within a modality, that advantage will disappear or reverse. A label-free shift score computed per pathway will predict which regime a new task falls into. The decisive evidence is the crossover between the two regimes, not a single leaderboard number.

Prior Work and Where This Sits

The mechanism comes from multi-axis blocked self-attention. Improved Transformer for High-Resolution GANs introduced it for generation, and MaxViT systematized it as a stack of three steps: a mobile convolution, then attention within non-overlapping $P\times P$ windows ("block"), then attention among tokens sampled at fixed intervals across the whole image ("grid"). MaxViT was developed on ImageNet and JFT-300M and has never been characterized with tiny cohorts. Related long/short splits appear in CrossFormer and CrossFormer++. We reuse the block/grid factorization and extend it to 3D. We do not claim the 3D extension as the contribution. It is the vehicle that makes the pathway question askable.

The closest existing work motivates the proposal by exposing the gap. NA-UNETR pretrains a model built from local sliding-window attention plus dilated neighborhood attention on 1,000 contrast CT angiography volumes from ImageCAS. It then LoRA-tunes the model on 20 non-contrast planning CTs. It applies adaptation without asking which pathway needs it, and it reports that the contrast gap remains unclosed. Transductive few-shot adapters adapt a frozen Swin-UNETR with 1 to 10 labeled scans. They use 3D convolutional adapters on the backbone's output features, plus an organ-size prior at test time. Because those adapters sit outside the backbone, they cannot choose among internal pathways. We differ from both by making the attention factorization itself the unit of adaptation, and by predicting in advance, from unlabeled data, which unit to adapt.

Two papers constrain the design. Akkaya et al. show that ViTs trained from scratch on small data fail for lack of locality. Adding multi-scale depthwise convolutions to the attention projections raised DeiT-Tiny on CIFAR-100 from 51.45% to 71.74%. This supports concentrating small-data learning in local operators. NA-UNETR's authors argue that unconstrained global attention can drown faint targets in background, which seems to cut against keeping a global pathway at all. We address this directly: one arm of the study removes the grid pathway at adaptation time, so the data can decide between "freeze global" and "drop global." HResFormer and MCANet show that local-global medical segmenters reach strong fully supervised results (89.10% and 83.52% Dice on Synapse). Neither measures behavior under scarcity, so they serve as fully supervised upper references rather than competitors.

Research Program

We first build a 3D block/grid encoder inside a UNETR-style decoder. Tokens are partitioned into local $P\times P\times P$ blocks and, separately, into a $G\times G\times G$ dilated grid, following a convolution step as in MaxViT. Parameters are tagged by pathway: convolution, block attention, grid attention, and decoder. We pretrain two source models. One is trained on ImageCAS for coronary arteries, and is the only public source matching the LAD setting. The other is trained on TotalSegmentator CT for multi-organ anatomy, matching the source used by Silva-Rodríguez et al.

Before adaptation, we compute a per-pathway shift score from unlabeled target scans. For layer $\ell$ and pathway $a$, let $\mu_s,\Sigma_s$ and $\mu_t,\Sigma_t$ be the mean and covariance of that pathway's output tokens on source and target scans. The score is

$$ d_{\ell,a}=\frac{\lVert \mu_s-\mu_t\rVert^2+\operatorname{tr}\!\left(\Sigma_s+\Sigma_t-2(\Sigma_s\Sigma_t)^{1/2}\right)}{\bar d^{\,\text{src}}_{\ell,a}}, $$

where the numerator is the Fréchet distance between the two feature distributions. The denominator $\bar d^{\,\text{src}}_{\ell,a}$ is the same distance between two random halves of the source set, so a score near 1 means "no more shift than sampling noise." Shift-guided adaptation then places a fixed LoRA budget on the pathways with the largest scores. Only the budget is fixed in advance; which pathway receives it is chosen by the score. This matters most at one shot, where there are no held-out labels for choosing among configurations.

The analysis has three parts. First, we test whether the shift score separates appearance-shift tasks from structure-shift tasks. Second, we test whether it ranks pathways in the same order as their measured adaptation benefit. Third, we compare pathway-wise adaptation against uniform and depth-wise adaptation, holding trainable parameters, tuning trials, and augmentation fixed.

Baselines

The do-nothing reference is the pretrained model applied zero-shot to the target, and, for new structures, the source model with only a new output head trained. This bounds how much any adaptation adds.

The strongest convolutional competitor is nnU-Net trained from scratch on the target scans with its own self-configuring pipeline. Its built-in locality is exactly the bias small data rewards. On the 20-scan non-contrast LAD task, nnU-Net reaches 42.54% Dice (NA-UNETR). We run it unmodified on identical splits, since handicapping it would make any win meaningless.

The strongest existing transfer method on the LAD question is NA-UNETR with LoRA. It reports 45.64% Dice and 38.16 mm 95th-percentile Hausdorff distance on LAD-SEG, with about 19.6M parameters. Beating it means exceeding 45.64% Dice and reducing HD95 below 38.16 mm on the same 20-scan protocol, under matched pretraining data (ImageCAS) and matched trainable-parameter count. LAD-SEG is a private institutional dataset, so this comparison depends on obtaining access through the authors. If access fails, the LAD arm is reported as not run, not approximated.

On the few-shot organ question, the strongest competitor is the spatial-adapter method of Silva-Rodríguez et al. on TotalSegmentator and FLARE'22 with $K\in\{1,\dots,10\}$. No reported per-shot Dice for their adapters was found on Lacuna. The file reports only that their size prior adds up to +0.9% Dice and that adaptation takes 4 to 8 minutes. We therefore rerun their method on our splits and report against our reproduction. We add their test-time size prior to every arm, so it does not confound the comparison.

Three ablations isolate the key ingredient. Uniform LoRA spreads the same budget over all pathways. Depth-wise adaptation tunes the last stages with the same budget. Random-subset adaptation tunes a parameter subset of matched size drawn without regard to pathway. Together these separate "pathway structure matters" from "fewer parameters help." We also run the same pathway-wise recipes on a Swin-UNETR backbone with shifted-window attention only, to check whether gains come from the 3D block/grid backbone or from the adaptation rule. Every arm gets the same number of hyperparameter trials, the same augmentation, and the same GPU budget. Each $K$ is averaged over at least five random draws of support scans, because at $K=1$ the variance between draws can exceed the differences between methods.

Why not just…?

Why not just train nnU-Net from scratch? It is a serious threat at $K\ge 10$. On LAD-SEG it trails NA-UNETR by only 3.1 points, and on full Synapse it reaches 86.99%, close to HResFormer's 89.10% (HResFormer). Its weakness should appear at $K\le 5$ and under appearance shift, where it cannot borrow anatomy from a large source. Our sample-efficiency curves will show where, if anywhere, the curves cross. If nnU-Net matches pathway-wise adaptation at all $K$, we will report that the pretrained global pathway does not earn its cost.

Why not just apply LoRA uniformly with a smaller rank? This is the real alternative. It is cheap, standard, and what NA-UNETR effectively does. Our claim survives only if pathway-wise adaptation beats it at identical trainable-parameter counts across several budgets, and only if that advantage reverses on structure-shift tasks. A win in both regimes would suggest generic regularization, not pathway semantics. In that case we would retract the semantic claim and keep only the empirical recipe.

Why not just freeze early layers and tune late ones? Depth-wise freezing encodes a different hypothesis: low-level features transfer and high-level ones do not. Under contrast or modality shift that hypothesis is backwards, since the low-level intensity statistics are exactly what changed. The depth-wise ablation at matched budget tests this directly.

Why not just use convolutional adapters on a frozen foundation model's outputs? Silva-Rodríguez et al. show that 3D spatial adapters beat MLP adapters. Their adapters, however, can only post-process features after the backbone has already applied source-domain local filters to unfamiliar intensities. If output-level adapters match internal block-pathway adaptation under appearance shift, then internal access is unnecessary, and our experiment will show this.

Why not just try both pathway choices and pick by validation? With 10 to 20 scans, leave-one-out selection is feasible and may make the shift score redundant. We therefore compare against oracle and leave-one-out selection. The score's claimed value is confined to $K\le 3$, where validation is not meaningful.

Proposed Experiments

The appearance-shift arm has two parts. The first transfers the ImageCAS-pretrained model to non-contrast LAD-SEG with 20 scans, pending access. The second transfers the TotalSegmentator CT model to abdominal MRI organs on ABD-MR with $K\in\{1,3,5,10\}$ volumes. For ABD-MR, RP-Net's 1-shot slice-level result of 79.26% Dice (RP-Net) is cited as context only, because its protocol (support–query matching without fine-tuning) differs from ours. The structure-shift arm follows the TotalSegmentator and FLARE'22 few-shot protocol for organs held out of pretraining, within CT. Metrics are Dice and HD95, standard deviation across support draws, trainable parameters, and adaptation time.

Three checks decide the outcome. First, under appearance shift, adapting only the convolution and block pathways must beat uniform LoRA at matched budget on at least one dataset. Second, under structure shift, that ordering must weaken or reverse. Third, the label-free shift score must rank pathways in the same order as their measured adaptation benefit across tasks, with rank correlation clearly above what random-subset controls give. If the first check holds but the second does not, the result reduces to "fewer, more local parameters help," which is already implied by Akkaya et al. If the grid-removal arm beats grid-freezing under appearance shift, NA-UNETR's skepticism about global attention in faint-target regimes is confirmed, and the global pathway should be dropped rather than preserved.