Adversarial Adaptation in Learned Anatomical Coordinates

When a segmentation model trained at one hospital is run on scans from another, its errors are often less about intensity and more about anatomy. The model fragments a kidney, leaves holes in a liver, or places an adrenal gland in the wrong spot, because it judged each voxel from local texture instead of from where that voxel sits in the body. We propose to fix this by changing where the adaptation happens, not by enlarging the network's view. First, we learn a correspondence network without any landmarks. It maps every point on any organ surface to a shared set of canonical "anatomical addresses," using only the labeled source masks. Then, on unlabeled target scans, a domain discriminator (a classifier that tries to tell source from target) compares the segmenter's predicted anatomy against real source anatomy address by address. The segmenter is trained to fool it.

In two sentences: the correspondence map gives every predicted surface point a global anatomical identity. A discriminator that reads those identities can therefore penalize anatomically wrong predictions anywhere in the body, without the memory cost of global attention over a 3D volume. If this works, the long-range context that local convolutions lack is supplied once, by a shape model trained on clean source labels. It is not something every target-domain layer must relearn.

Core Claim

Adversarial alignment works best in a compact space (Synergistic Image and Feature Adaptation aligns predicted masks and reconstructed images rather than raw features, because high-dimensional feature alignment is unstable). We claim that the right compact space for cross-scanner segmentation is the set of predicted organ surfaces expressed in landmark-free correspondence coordinates. Concretely, we borrow the shape-to-universe idea of Unsupervised Deep Multi-shape Matching, which maps each shape to $d$ abstract reference indices so that all pairwise matches are cycle-consistent. We train it on the joint multi-organ surface cloud built from source ground truth.

The discriminator then sees, for each universe index, the predicted local geometry and inter-organ offsets. It is asked whether the anatomy as a whole looks like real source anatomy. The new element is not adversarial adaptation and not shape priors, both of which exist. It is the use of dense, learned correspondence as the coordinate system of the adversarial game. This turns a local, texture-sensitive alignment signal into a global, anatomy-indexed one.

Relation to Prior Work

The closest work is Learning Deep Features for Shape Correspondence with Domain Invariance. It flattens 64×64 geodesic surface patches into 2D images, trains a Siamese network to match them, and uses a gradient reversal layer to make the features invariant between bone types (femur to pelvis). It reports 0.80 AUC on scapula correspondence. It points the opposite way from us. It uses adversarial training to make correspondence features domain-invariant, works on local patches without global context, and never touches scanner shift in images. We use correspondence to make segmentation domain-invariant, and the correspondence is global by construction.

The second neighbor, Unsupervised Domain Adaptation through Shape Modeling, constrains target predictions with a variational autoencoder that reconstructs plausible binary pancreas masks. It raises MSD Dice from 0.6777 to 0.7574. It motivates our premise that shape is the stable invariant across scanners, but it constrains us in two ways. It already captures holistic mask plausibility, so our gains must come from what a voxel autoencoder handles poorly: thin and small structures, multi-organ layout, and symmetric pairs. It also handles only one organ.

Several works supply the method. The universe formulation comes from Unsupervised Deep Multi-shape Matching, which reaches mean geodesic error 1.5 on FAUST against 1.7 for Deep Shells. Its stated requirement of at least one complete shape is met here because source ground-truth masks are complete. The point-cloud alternative to meshes comes from CorrNet3D, which learns dense correspondence from raw points with a DGCNN encoder and fast non-iterative matching. Predicted target surfaces are noisy and not watertight, so we cannot rely on Laplace–Beltrami spectra.

CorrNet3D also reports a limitation that motivates the multi-organ design: local-geometry correspondence confuses symmetric parts. Left and right kidneys are nearly mirror images in isolation, but they are unambiguous given the liver and spleen. The receptive-field literature constrains the efficiency argument. Dense non-local blocks (Compact Generalized Non-local Network) scale quadratically in voxels. CCNet and hybrid models like ViT-V-Net reduce that cost. Even so, AMOS finds a plain 3D U-Net (89.04% mDice) beating Swin-UNETR (86.32%) on CT. Global attention in the backbone is therefore not obviously the cure. We move global reasoning to a few thousand surface points instead.

Research Program

The first component is the correspondence model. From source ground-truth masks we extract surfaces of all organs and sample a joint point cloud of roughly 4–8k points, with each point tagged by organ label. We train a DGCNN-based encoder with a global pooling branch to assign each point a soft distribution over $d$ universe indices. Training is unsupervised, using cross-reconstruction (Chamfer) and neighborhood-consistency losses as in CorrNet3D, with cycle consistency supplied by the universe construction. We sweep $d$ and report its effect, since the file flags fixing $d$ as an open limitation.

The second component is the discriminator in universe coordinates. For a target scan, the segmenter's softmax output defines a weighted point set: candidate boundary voxels weighted by predicted probability, so that gradients reach the segmenter through the weights. The frozen correspondence model assigns universe indices. For each index we pool predicted position relative to the body centroid, local normal, and distance to the nearest other organ. This yields a fixed-size $d \times k$ anatomical descriptor. A small discriminator compares these descriptors from target predictions against those from source ground truth. The segmenter minimizes its source supervised loss plus $\lambda$ times the adversarial loss on target. A per-index discriminator output gives an interpretable map of where target anatomy is implausible.

The third component is analysis. We will report where the adversarial signal concentrates, whether it fires on fragments and swapped organs, and how often the correspondence itself fails on badly broken predictions. A self-reinforcing failure is possible: wrong anatomy could be assigned wrong addresses that look plausible. We will measure it by injecting known defects into source masks, in the style of ErrorNet, and checking whether the discriminator flags them.

Baselines

The naive reference is a 3D U-Net trained on source and applied to target without adaptation. This is the setting where AMOS reports drops of over 20% on unseen MRI scanners; the file gives no exact per-scanner value.

The strongest existing method on the shared pancreas setting is the shape-modeling teacher–student method of Unsupervised Domain Adaptation through Shape Modeling, which reports MSD Dice 0.7574 after adaptation from 0.6777. We will rerun it with the same backbone, data splits, and hyperparameter search budget as our method rather than copy numbers. Beating it means exceeding its rerun Dice on MSD and the other target sets, and clearly reducing boundary error (95th-percentile Hausdorff distance) and connected-component errors.

The second strong competitor is output-space adversarial alignment as in SIFA. It is the most direct "compact-space adversarial" alternative. We run it with an identical segmenter and discriminator capacity matched in parameters. The file reports no SIFA number on these benchmarks, so we have no reported result to beat and will compare against our rerun.

We add frequency amplitude normalization (Yu et al.) combined with heavy intensity augmentation. This is a non-adversarial input-side baseline.

Two ablations isolate the key ingredient. The first uses the same point-cloud discriminator on predicted surfaces with no universe coordinates, a plain DGCNN reading raw points. This tests whether correspondence, not merely point geometry, carries the signal. The second replaces the global correspondence model with a patch-local variant that has no global pooling, mimicking the local-patch design of the closest prior work. All methods share data, epochs, and a fixed tuning budget per method.

Why not just…?

Why not just use a voxel shape autoencoder prior? This is the real threat, and it is already shown to work on pancreas. Our bet is that a voxel autoencoder reconstructs coarse blobs well but smooths away small and thin structures, and has no explicit notion of which surface part is which. Correspondence coordinates penalize a misplaced adrenal gland or a kidney swap directly. If on multi-organ AMOS our gains over the autoencoder baseline do not concentrate on small organs and symmetric pairs, the correspondence machinery is not earning its cost. We would then report that.

Why not just align segmentation masks adversarially, as SIFA does? A mask-level discriminator is a convolutional network over volumes, so it inherits the same local receptive field that caused the errors. It sees plausibility patch by patch. The no-universe and local-only ablations test exactly this. If a raw point or mask discriminator matches our method, the correspondence coordinates are unnecessary.

Why not just put global attention in the backbone, or augment harder? AMOS shows a transformer backbone underperforming a 3D U-Net on CT, and attention does not address the target data never being seen. Strong augmentation and amplitude normalization may close much of the CT gap, which is why we include them. However, MRI intensities lack physical units, and the >20% unseen-scanner MRI drop in AMOS suggests intensity tricks alone leave a large residual. That residual is where we expect our method to matter.

Proposed Experiments

The primary benchmark is AMOS. We run CT-to-MRI adaptation on 15 organs and, more importantly, MRI adaptation to scanners held out from training. Metrics are per-organ Dice, 95th-percentile Hausdorff distance, and a topology score that counts extra connected components and holes per organ.

The secondary benchmark is the pancreas setting of the shape-modeling paper: NIH as source, with MSD and Synapse as targets. This allows a direct comparison with its reported 0.7574 MSD Dice under our rerun.

As a sanity check on the correspondence module, we evaluate it on FAUST against the 1.5 geodesic-error figure from Unsupervised Deep Multi-shape Matching. We also run a symmetry test on paired organs: the fraction of left-kidney points mapped to right-kidney universe indices.

Three decisive checks would make the case. The first is that the method beats the rerun shape-autoencoder baseline on held-out-scanner AMOS MRI, with the gain concentrated on small organs (adrenals, duodenum, gallbladder) and on topology errors. The second is that the no-universe ablation loses most of that gain. The third is that, in the defect-injection test, the per-index discriminator flags injected anatomical errors at a clearly higher rate than a mask-level discriminator. If the first holds but the second fails, the contribution reduces to "surface point clouds are a good adversarial space," which would be a smaller but still reportable result.