Scanner Shift as a Separate Direction: Orthogonal Organ Subspaces for Cross-Scanner Segmentation

When a 3D organ segmentation model trained on CT, or on one hospital's MRI scanners, meets a new MRI scanner, accuracy can fall by more than 20% in Dice overlap. Small organs such as the adrenal glands and duodenum degrade most (AMOS). This proposal tests a simple geometric explanation. Inside the network, a voxel's feature vector has no designated place for scanner appearance to go. A change in contrast or noise therefore moves features along the same directions the classifier uses to tell organs apart, and small organs, whose feature regions are thin and crowded by large neighbors, get pushed across decision boundaries first. The fix we propose is to give each organ its own multi-dimensional subspace, keep those subspaces mutually perpendicular, and reserve one extra subspace, perpendicular to all organs and ignored by the classifier, for appearance variation. Training then requires that when an image is restyled to mimic another scanner, its voxel features may move, but only within that reserved "nuisance" subspace.

The same geometry yields a free reliability signal at test time. A voxel whose feature cannot be explained by its predicted organ's subspace plus the nuisance subspace is showing a change the model has never seen. This leftover component can drive abstention and calibration without a separate post-hoc network. In two sentences: scanner changes should move features sideways into a reserved direction the classifier ignores, never toward another organ. Training for this should help small organs most under unseen-scanner shift, and the part of a feature that fits nowhere tells you when not to trust the output.

Core Claim

Our claim is that cross-scanner robustness in dense multi-organ segmentation is limited less by how much features change under shift than by which direction they change in. Two existing approaches miss this in opposite ways. Forcing features to be fully invariant, or collapsing each class across domains into a single ray as Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization (LDDG) does through rank regularization, removes legitimate anatomical variation along with scanner variation. Pushing class features apart, as Orthogonal Projection Loss (OPL) does with an inter-class cosine penalty, never says where scanner variation should go. We make a specific bet. Allow each organ a subspace of several dimensions, following the maximal coding rate reduction argument that classes should span rich, mutually orthogonal subspaces rather than collapse. Add an explicit nuisance subspace, and constrain the direction of augmentation-induced feature displacement rather than its size. We predict this beats both full invariance and plain class orthogonality on held-out scanners, with the largest gains on small organs.

How This Differs From Prior Work

The closest work falls into three groups, and each leaves a gap this proposal fills. The first group imposes orthogonality in the wrong space or setting. Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation (FedIAT) keeps hospital-specific low-rank weight updates perpendicular to shared ones when fine-tuning a segmentation foundation model across hospitals. That orthogonality lives in parameter space, needs a federation of clients, and was tested on 2D histology and fundus images. We act on voxel features, in a single centralized training run, on 3D volumes. Unsupervised Domain Adaptation in Semantic Segmentation via Orthogonal and Clustered Embeddings does make class centroids orthogonal in pixel features. However, it uses pseudo-labels on unlabeled target images and was evaluated only on synthetic-to-real driving scenes. We use no target data, which rules out the confirmation loop its authors acknowledge. OPL works at the whole-image level on natural-image benchmarks and has no notion of a nuisance direction.

The second group supplies our mechanism and also constrains it. LDDG motivates the problem: it shows that geometric structure in features helps medical domain generalization, reporting 0.7881 Dice versus 0.7425 on spinal cord gray matter MRI. Its rank-$C$ constraint, however, collapses each class to one direction. Maximal coding rate reduction argues that this kind of collapse discards useful diversity, and that argument is why we keep class subspaces multi-dimensional. Task-Agnostic Undesirable Feature Deactivation Using Out-of-Distribution Data (TAUFE) pushes features of auxiliary out-of-distribution images to zero so that nuisance features switch off. Medical imaging has no clean auxiliary dataset that carries scanner artifacts without anatomy. We replace that dataset with paired restylings of the training images themselves. Let OOD Feature Exploring Vast Predefined Classifiers reserves an orthogonal null space for outlier classes. Our nuisance subspace is related, but it holds shifted versions of known classes, not alien inputs.

The third group is the calibration literature. Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift and Improved post-hoc probability calibration for out-of-domain MRI segmentation show that shifted inputs produce confident errors, and both repair confidence after training. The first needs out-of-domain validation data. The second needs a learned shape prior, which its authors note may fail on irregular structures. We test whether a geometric residual computed from the trained model alone gives comparable referral quality.

Method

Let $f(x)_v \in \mathbb{R}^d$ be the decoder's penultimate feature at voxel $v$. We keep orthonormal bases $U_c \in \mathbb{R}^{d \times d_c}$ for each organ $c$ (including background) and a nuisance basis $N \in \mathbb{R}^{d \times d_n}$. The total dimension $\sum_c d_c + d_n \le d$; with AMOS's 15 organs plus background and $d=256$, there is room for $d_c \approx 8$–$12$ and $d_n \approx 64$. The classifier reads only the projections onto the organ subspaces, so the nuisance subspace lies in its null space by construction, as in OSLNet's architectural orthogonality. The bases are learned parameters, kept orthonormal with a QR (orthogonal-triangular) reparametrization, which makes the organ and nuisance subspaces mutually perpendicular exactly rather than through a penalty.

Each training volume $x$ is paired with a restyled copy $\tilde{x}$. The restyling uses Fourier amplitude swapping, which changes contrast and intensity while keeping structure (FedDG), along with random nonlinear intensity remapping and simulated noise and bias fields. Let $\delta_v = f(\tilde{x})_v - f(x)_v$ be the feature displacement at voxel $v$. Let $P = \sum_c U_c U_c^\top$ be the projector onto the organ subspaces and $Q = NN^\top$ the projector onto the nuisance subspace. The auxiliary loss is

$$ \mathcal{L}_{geo} = \frac{1}{|V|}\sum_{v}\Big(\|P\,\delta_v\|^2 + \lambda\,\|(I - U_{y_v}U_{y_v}^\top - Q)\,f(x)_v\|^2\Big), $$

where $y_v$ is the ground-truth organ at voxel $v$ and $\lambda$ weights the second term. The first term allows displacement only along nuisance directions. The second term requires each voxel feature to lie in its own organ's subspace plus the nuisance subspace, which keeps organs apart without collapsing any organ to a single ray. The full objective is cross-entropy plus Dice plus $\mathcal{L}_{geo}$. Voxels are sampled in a class-balanced way so that small organs contribute equally.

At test time, the residual $r_v = \|(I - U_{\hat{y}_v}U_{\hat{y}_v}^\top - Q) f(x)_v\| / \|f(x)_v\|$ is computed for the predicted organ $\hat{y}_v$. This residual serves as the abstention score and as a per-voxel temperature for calibration.

Research Program

The program has three parts.

The first part is a measurement study, done before any new training. Using a standard 3D-UNet trained on AMOS CT, we compute, for each organ, the shift in class-conditional mean feature between CT and each MRI scanner group. We then measure what fraction of that shift lies along the directions separating that organ from its anatomical neighbors. If scanner-induced movement is already mostly perpendicular to those directions, the geometric story is wrong and we stop. If it is concentrated along them, and more so for small organs, the premise holds.

The second part trains the method and its ablations under a leave-scanner-out protocol.

The third part tests whether the residual $r_v$ predicts errors on unseen scanners better than softmax entropy and post-hoc calibration do.

A secondary question is whether one nuisance subspace suffices, or whether separate nuisance subspaces in the encoder and decoder work better. FedIAT reports that encoders mostly see appearance shift while decoders see labeling-convention shift. We treat this as a later extension, not the core claim.

Baselines

The naive reference is a well-tuned 3D-UNet trained with cross-entropy and Dice and standard intensity augmentation. This matters because AMOS found that a plain 3D-UNet (89.04% mean Dice on CT) beat Swin-UNetr (86.32%). We use that 3D-UNet as the shared backbone for every method. All methods get the same patch size, epochs, and a fixed hyperparameter-search budget of equal trials, with model selection done on held-out source scanners only, never the target.

The strongest existing competitor is FedDG's recipe run centrally on pooled source data: Fourier amplitude interpolation plus boundary-oriented contrastive episodic learning (FedDG). FedDG reports leave-one-site-out Dice of 87.39% on six-site prostate MRI, versus 85.57% for federated averaging. It is the strongest competitor because it targets exactly the appearance shift we address and uses the same Fourier augmentation, so any gain we show cannot be credited to the augmentation alone. On AMOS's unseen-MRI-scanner setting, no reported result for this method was found on Lacuna, so we run it ourselves. On AMOS, AMOS reports a drop of over 20% on unseen MRI scanners but no per-method unseen-scanner Dice was found on Lacuna, so all AMOS numbers come from our runs under the shared protocol.

LDDG's rank regularizer, ported to voxel features, is the second geometric baseline. Its reported spinal cord result of 0.7881 Dice gives a second benchmark on which we can compare directly. The third geometric baseline is a voxel-level OPL with orthogonal centroids in the style of the driving-scene adaptation work, without pseudo-labels.

Three ablations isolate the key ingredient. Removing the nuisance subspace and penalizing $\|\delta_v\|^2$ directly tests direction-only versus full invariance. Setting $d_c = 1$ reproduces LDDG-like collapse. Keeping the augmentation pairs but applying consistency only to output probabilities tests whether feature geometry matters at all. For calibration, the baselines are softmax entropy and the local temperature scaling with shape prior from Improved post-hoc probability calibration. On MS-CMRSeg, that paper reports cross-sequence expected calibration error of 15.37%, down from 16.78%.

We count the proposal as beating the field if it does two things. First, it must exceed the centralized FedDG recipe on mean Dice for unseen AMOS MRI scanners, with a gain over the full-invariance ablation concentrated in the small-organ group, stable across three seeds. Second, its residual-based expected calibration error on MS-CMRSeg must be at or below 15.37% without any shape prior.

Why not just…?

Why not just use Fourier style randomization with a consistency loss? This is the most serious threat, and the output-consistency ablation exists to answer it. FedDG itself found amplitude interpolation insufficient without its additional boundary-contrastive episodic learning, which suggests augmentation alone leaves accuracy on the table. Output consistency still constrains only the logits. A feature can move toward a neighboring organ and stay just inside the boundary, which is exactly the fragile case for thin structures. If output consistency matches our method on small-organ Dice under unseen scanners, the geometric claim is refuted and we will report it that way.

Why not just make features fully invariant to restyling? Full invariance resembles LDDG's collapse: it removes the information that distinguishes real anatomical variation from appearance change, and maximal coding rate reduction argues this costs robustness. The $\|\delta_v\|^2$ ablation tests this directly. If it ties our method, the nuisance subspace is unnecessary, and the contribution reduces to class orthogonality plus augmentation.

Why not just add OPL to segmentation? OPL separates class directions but gives scanner variation nowhere to go. On natural images it reported only whole-image gains, such as ImageNet top-1 accuracy rising from 76.15% to 76.98%. The voxel-OPL baseline measures how much of any gain comes from orthogonality alone versus the reserved nuisance subspace.

Why not just calibrate after training? Post-hoc methods change confidence, not segmentation quality. They also require either out-of-domain validation data (Avoiding Catastrophic Referral Failures) or a shape prior that may not fit irregular anatomy. If local temperature scaling clearly beats our residual on calibration, we will keep only the accuracy claim.

Proposed Experiments

The main experiment is on AMOS. We train on all CT plus MRI from a subset of scanner groups and test on held-out MRI scanner groups, using scanner metadata where the release provides it. If it does not, we fall back to CT-only training with MRI testing. We report mean Dice and 95th-percentile Hausdorff distance per organ, grouped into large organs (liver, spleen) and small organs (adrenal glands, duodenum, and similar). Replication runs on six-site prostate MRI, leave-one-site-out, compared against FedDG's 87.39%, and on spinal cord gray matter, compared against LDDG's 0.7881.

Calibration and referral are tested on MS-CMRSeg, training on bSSFP and testing on LGE. We report expected calibration error and the area under the error-versus-referral-rate curve.

The decisive checks are three. First, the pre-training measurement must show that cross-scanner feature drift projects onto inter-organ directions in the baseline model. Second, in our model, the share of real cross-scanner class-mean shift that falls inside the nuisance subspace should rise substantially. This must hold on genuinely unseen scanners, not only on the synthetic restylings used in training. Otherwise the model has only memorized the augmentation family. Third, the small-organ Dice gain over the full-invariance and output-consistency ablations should exceed seed variance. If the second check fails while Dice still improves, the improvement is not happening for the reason we claim, and the note's central hypothesis should be considered false.