Vision transformers split an image into patches and let every patch attend to every other patch. They have no built-in sense that neighbouring pixels belong together, so on small medical datasets and on scans from unfamiliar machines they often lose to plain convolutional networks. This proposal tests whether one number explains that failure, and whether training on that number can repair it. For a single image, we build a similarity matrix among a network's patch features and ask what fraction of the ground-truth segmentation mask lies in that matrix's few dominant directions. In learning theory this quantity is called kernel-target alignment. It measures whether the network already "sees" the organ as a coherent group of patches.
We claim three things. First, this within-image alignment, computed on the training hospital's data before or early in training, predicts which architectures will degrade on unseen scanners. Second, the degradation on unseen scanners shows up as label energy leaking out of the dominant directions, and it is worst for small structures. Third, a loss that keeps label energy in the dominant directions of the feature similarity matrix, computed on physically perturbed "fake scanner" views, gives a plain transformer the spatial grouping it lacks. This loss needs no second guide network and no target-hospital data.
Core Claim and What Is New
Three bodies of work bear on this question, and none of them measures alignment inside a segmentation image or under scanner shift. Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks shows that in two-layer networks, labels concentrated on the top eigenvectors of the network's kernel are fit quickly and generalize well. It also gives a data-dependent complexity $\sqrt{y^\top (H^\infty)^{-1} y}$. That result comes from the lazy-training regime, classifies across samples, and involves no shift. We borrow the quantity and move it inside a single image.
Representation Alignment in Neural Networks measures alignment between hidden-layer singular vectors and labels across a dataset. It links a collapse in alignment to negative transfer. This motivates our shift hypothesis, but that work never looks at dense prediction or at scanners.
Training the Untrainable: Introducing Inductive Bias via Representational Alignment transfers architectural priors by aligning a target network's representations to a frozen, even randomly initialized, guide network using centered kernel alignment. It is the closest method to ours. It differs in three ways: it matches another network rather than the labels, it pays the memory cost of two networks' activation Gram matrices, and it is evaluated only in-distribution on non-medical tasks. The obvious combination, "Guidance with a random 3D CNN guide applied to a medical ViT", is therefore a baseline here, not the contribution.
Our quantity is defined as follows. Let $F \in \mathbb{R}^{n \times d}$ be the centered patch features of one image at a chosen layer, with $n$ patches and feature width $d$. Let $u_1,\dots,u_k$ be the top-$k$ left singular vectors of $F$, which are the dominant eigenvectors of the patch similarity matrix $K = FF^\top$. Let $y_c \in \mathbb{R}^n$ be the centered indicator of class $c$ in the downsampled mask. The within-image alignment is
$$A_k(F, y_c) = \frac{\sum_{i=1}^{k} (u_i^\top y_c)^2}{\|y_c\|_2^2},$$
which is the fraction of the class-$c$ mask energy captured by the network's $k$ dominant feature directions. We also report the ridge analogue of the Arora et al. measure, $y_c^\top (K + \lambda I)^{-1} y_c$, where $\lambda$ is a small ridge constant. Both scores are computed from a truncated singular value decomposition of $F$, which costs $O(nd^2)$ and never forms the $n \times n$ matrix. This removes the memory bottleneck noted for CKA guidance.
Research Program
The first strand is diagnostic. We train a family of segmentation models on a single source domain: a plain ViT encoder with a UNETR-style decoder, Swin UNETR (Swin UNETR), a CvT-style convolutional-projection transformer, 3D-UNet/nnUNet, and TransNetR (TransNetR) for the endoscopy setting. Each model is trained both from random initialization and from pretrained weights. We record $A_k$ per class and per layer at initialization, after 1% of training, and at convergence. We then measure it on held-out source images and on each unseen target domain; target labels are used only for this analysis.
The hypothesis is sharp. $A_k$ measured early on source data should rank the architectures by their final out-of-distribution Dice better than two cheap alternatives: source validation Dice and parameter count. It should also rank them by the size of their out-of-distribution drop.
The mechanistic prediction for AMOS, which reports a drop of over 20% on unseen MRI scanners (AMOS), is more specific. The per-organ drop in Dice should track the per-organ drop in $A_k$. The largest alignment losses should fall on the adrenal glands and duodenum, the structures AMOS identifies as fragile.
The second strand is the intervention. During source training we generate a "fake scanner" view of each image. For MRI this uses random nonlinear intensity remapping, bias fields, and resolution changes; for CT and endoscopy it uses windowing and color shifts. The loss is the usual Dice plus cross-entropy, plus $\beta \sum_c (1 - A_k(F', y_c))$, where $F'$ holds the perturbed view's features at two or three encoder layers and $\beta$ weights the added term.
The key ingredient is the restriction to the top-$k$ subspace. The loss does not ask all features to be invariant, and it does not ask all same-class patches to be similar. It asks only that the mask be expressible by the few directions that dominate the representation, even after the intensity statistics change. Everything outside those directions is left free for the decoder to use.
Gradients flow through the truncated decomposition. We monitor the eigengap between the $k$-th and $(k{+}1)$-th singular values and fall back to a projection with a smooth weighting of the singular values when the gap closes. We sweep $k$ per dataset on source validation only.
Baselines
The naive reference point is empirical risk minimization, meaning ordinary supervised training with standard augmentation. It is applied to the same plain ViT-UNETR, with an identical schedule and identical augmentation. This tells us how large the transformer deficit is before any fix.
The strongest existing method on the CT setting is a well-tuned convolutional network. AMOS reports 89.04% mDice for 3D-UNet on its CT test set, versus 86.32% for Swin-UNetr (AMOS). We run nnUNet and Swin UNETR with their published training recipes on our splits.
On polyps, the strongest competitor is TransNetR. It reports Dice 0.8706 and mIoU 0.8016 on Kvasir-SEG, and a 2.81% mIoU improvement over the next best model on PolypGen, although no absolute PolypGen value is reported in the file. The DPN68×2 double encoder-decoder reports 76.19% Dice on the external ETIS-LaribPolypDB set (Double Encoder-Decoder Networks).
On retina, the reference is ErrorNet, which reports an average cross-dataset Dice of 71.0 when trained on CHASE (ErrorNet). On the small mammogram task, the reference is the winning EfficientNet pipeline at Dice 0.837, trained with under 8 H100-hours (Benchmarking Transfer Learning for Dense Breast Tissue Segmentation).
We re-run every baseline ourselves on identical splits. Published numbers serve only as sanity checks that our reproductions are faithful.
The closest method baseline is Guidance (Training the Untrainable). It aligns the ViT's layer representations by centered kernel alignment to a frozen, randomly initialized 3D CNN or U-Net guide. We run it both with and without the same perturbed views, so that it sees exactly the data our method sees.
Three ablations isolate the ingredient. The first is augmentation only: the perturbed views are added as training data, with no alignment term. The second is a full-kernel supervised affinity loss, which pulls same-class patch features together on the perturbed views; this is our loss with $k=n$. The third is our loss on clean views only, with no perturbation.
All methods share the architecture where applicable, the data, the number of epochs, the GPU budget, and an equal hyperparameter search of 16 configurations each, all selected on source validation only. To count as beating a baseline, the method must achieve a higher mean out-of-distribution Dice across target domains, with confidence intervals from three seeds and a bootstrap over cases that exclude zero. It must also keep source Dice within one point.
Why not just…?
Why not just use heavier or physics-based augmentation? Augmentation alone is the most likely threat, and it is exactly our first ablation. Our loss uses the same perturbations, so any gain it adds over augmentation-only training is attributable to the spectral term. The file offers some reason to expect augmentation to plateau: generic augmentation and generic self-supervision gave negligible or negative gains on the 596-image mammogram set (Benchmarking Transfer Learning). If augmentation-only matches our method out of distribution, we will report the intervention as unnecessary. The diagnostic strand stands on its own.
Why not just use nnUNet and skip transformers? On AMOS CT this is currently the better choice (89.04% versus 86.32% mDice). We do not assume transformers must win. Instead we apply the same loss to nnUNet. If it narrows nnUNet's own out-of-distribution drop, the finding is about alignment under shift, not about rescuing ViTs. If the regularized ViT still trails nnUNet out of distribution, we will say so. The contribution then becomes the pre-training predictor and the mechanistic account of the drop.
Why not just use Guidance with a random CNN guide? Guidance matches the guide's geometry, including the guide's own sensitivity to intensity. It is also reported to inherit the guide's errors. Our loss matches the labels. The head-to-head comparison at equal data and compute settles which works better. The memory comparison, a second network's activations versus a $d$-dimensional decomposition, is reported alongside it.
Why not just use a supervised pixel-affinity or contrastive loss? This is our $k=n$ ablation. The claim is specific: forcing invariance on all directions over-constrains the representation and hurts source accuracy, while restricting to label-aligned dominant directions does not. If $k=n$ performs as well as small $k$, the spectral framing adds interpretability but not accuracy. We would then report the diagnostic as the main contribution.
Why not just fine-tune a foundation model with low-rank adapters? The mammogram benchmark found MedSAM2 and DINOv3 below CNNs, and parameter-efficient updates below full fine-tuning. We include MedSAM2 with full fine-tuning as a secondary baseline on the polyp and mammogram settings. We also measure its $A_k$, which lets us test whether the predictor explains foundation-model failures too.
Proposed Experiments and Decisive Checks
The data settings are as follows. On AMOS we run two splits: CT to MRI, and MRI with whole scanners held out. On endoscopy we train on Kvasir-SEG and test on PolypGen, BKAI-IGH, CVC-ClinicDB and ETIS. On retina we train on CHASE and test on DRIVE, STARE, ARIA and HRF. M&Ms multi-vendor cardiac MRI is an additional vendor-shift test; no reported result found on Lacuna for M&Ms. The mammogram set tests the data-scarcity regime without shift. Metrics are per-class Dice, boundary F1, and expected calibration error. The drop from in-distribution to out-of-distribution performance is the primary quantity.
Three checks decide the proposal. First, prediction: across roughly twenty architecture and initialization combinations, the Spearman correlation between early source $A_k$ and out-of-distribution Dice must clearly exceed that of source validation Dice. If it does not, the diagnostic claim fails. Second, mechanism: on AMOS unseen MRI scanners, the per-organ Dice drop must be correlated with the per-organ drop in $A_k$, and the correlation must survive controlling for organ size. Otherwise alignment is merely a symptom of small-structure difficulty. Third, intervention: the regularized ViT must beat both augmentation-only training and Guidance out of distribution, and small $k$ must beat $k=n$.
The three checks can fail independently, so each outcome is informative. A positive result on all three would give a label-driven, single-network way to instill spatial grouping in transformers that holds up across scanners. It would also explain in spectral terms why local structure matters most for the organs that break first.
References
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Representation Alignment in Neural Networks
- Training the Untrainable: Introducing Inductive Bias via Representational Alignment
- Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation
- ErrorNet: Learning Error Representations from Limited Data to Improve Vascular Segmentation
- Benchmarking Transfer Learning for Dense Breast Tissue Segmentation on Small Mammogram Datasets