Segmenting Small Organs Under Scanner Shift by Attending to the Shapes of Their Neighbors

When a segmentation network trained on one hospital's scans is deployed on another hospital's scanner, it does not fail evenly. The AMOS abdominal benchmark shows that models lose more than 20% of their accuracy on MRI from scanner manufacturers they never saw during training. The same benchmark shows that large, high-contrast organs such as the liver and spleen hold up well, while small, irregular organs (pancreas, adrenal glands, duodenum) are where accuracy collapses (AMOS). That asymmetry suggests an approach. Under a scanner change, the image texture around the pancreas becomes unreliable, but the pancreas still sits in a fairly stable place relative to the splenic vein, the left kidney, the aorta and the duodenal curve. Clinicians use exactly this cue when the pancreas is hard to see.

We propose to make the network use the cue explicitly. We borrow a mechanism from hand–object pose estimation, where a hand's hidden fingers are recovered by letting them "look at" the visible object's geometry through cross-attention. Cross-attention is an attention layer in which one set of features (the queries) reads information from a different set (the keys and values). In our version, the small target organ's queries do not read image features of its neighbors. They read the predicted shapes of its neighbors, encoded as signed distance fields (each voxel stores how far it lies inside or outside an organ's boundary). These shapes contain no image intensities, so the context they supply is largely insensitive to scanner contrast by construction. Each neighbor's influence is scaled by how stable that neighbor's own prediction is under intensity perturbation. When an anchor organ is itself confused, it is consulted less.

Core Claim

Under acquisition shift, the errors on small abdominal organs come mostly from local intensity evidence going bad, not from changes in anatomy. A module that conditions the small organ's prediction on intensity-free geometry of reliably segmented neighbors, weighted by measured reliability, should recover a substantial part of the small-organ accuracy lost on unseen scanners. It should do so beyond what intensity augmentation and ordinary inter-class attention already provide. The falsifiable part is the comparison with a matched module whose keys are image features rather than neighbor shapes. If the two perform the same, the claim is wrong.

Relation to Prior Work

The mechanism comes from interaction modeling. HOISDF predicts signed distance fields for hand and object and lets query points from one entity attend to the other's distance field. It reports 10.1 mm mean joint error on DexYCB. Semi-Supervised 3D Hand-Object Poses Estimation With Interactions in Time shows that exchanging features at contact regions lets known object geometry resolve occluded hand joints. Learning Human Mesh Recovery in 3D Scenes uses cross-attention to align a body mesh to scene geometry. We reuse the idea that one entity's geometry, rather than its appearance, constrains another. We change three things relative to this work. Our "occluder" is a domain shift rather than a physical occlusion. Our anchor geometry is itself predicted and possibly wrong, which is why we add a reliability weight. Our entities are organs with stable but non-contacting spatial relations.

Two papers come closest to our intersection. CCQ already argues that organs should not be treated independently. It models inter-organ dependencies with self-attention among class queries and uses cross-attention skip connections, reaching 76.41% average Dice on MOTS. Its queries read image features, however, and it is evaluated only within one CT distribution, so it never tests whether inter-organ relations survive a scanner change. In our view, attending to image features is exactly what breaks under shift. Human-Centric Transformer for Domain Adaptive Action Recognition does combine entity–context cross-attention with domain shift, decoupling human dynamics from context prototypes. It is built for video classification and needs unlabeled target data for adversarial alignment, whereas we target dense 3D segmentation with no target data at all.

The medical domain-generalization work supplies both our strongest baseline and our design constraints. Causality-Inspired Single-Source Domain Generalization simulates new scanners with random shallow intensity networks (GIN) and breaks foreground–background shortcuts (IPA), reaching 86.31 Dice on abdominal CT→MRI. It works entirely in input space, so we train our base network with it rather than compete against it. Domain Game separates anatomy features from style features, but it treats anatomy as one monolithic code for a single target. Train Once, Deploy Anywhere (EGSDG) feeds Canny/Sobel edge maps as extra input channels. Its authors note that handcrafted edges become noise on small, low-contrast targets. This motivates us to take geometry from predicted neighbor shapes rather than from filters.

Global shape priors constrain the design from another direction. ErrorNet uses a VAE shape prior to repair vessel segmentations. Improved post-hoc probability calibration uses a denoising-autoencoder shape prior plus test-time intensity jitter to rescale confidence. Its authors report that such priors work for regular shapes like cardiac ventricles but not for variable multi-organ anatomy. Our module avoids a single global prior and instead learns relations conditioned on neighbors. We reuse their test-time jitter idea as our reliability signal.

Research Program

The system has two stages.

The first stage is a standard 3D U-Net that segments all AMOS organs. It is trained on source CT with GIN+IPA augmentation, so any gain we report is on top of the strongest augmentation recipe. From its output we compute signed distance fields $\phi_k$ for a fixed set of anchor organs (liver, spleen, both kidneys, aorta, inferior vena cava, stomach). The anchors are chosen as the organs with high fidelity in AMOS.

For each anchor we compute a reliability score $r_k \in (0,1]$. It is the mean voxelwise Dice agreement of that anchor's mask across $M$ test-time intensity-jittered copies of the input. We use this score because it measures sensitivity to contrast changes, which is the failure mode under scanner shift.

The second stage refines each target organ (pancreas, adrenals, duodenum, gallbladder, esophagus). A query at voxel $x_i$ inside a coarse region of interest attends to tokens sampled from the anchors:

$$ a_i=\sum_j \operatorname{softmax}_j\!\Big(\tfrac{q_i^\top W_K\, g_j}{\sqrt d}+\log r_{k(j)}\Big)\,W_V\, g_j,\qquad g_j = \mathrm{MLP}\big(\phi_{k(j)}(x_j),\ x_j-x_i,\ e_{k(j)}\big). $$

In this equation, $q_i$ is the query embedding of voxel $x_i$. Token $j$ is a point $x_j$ sampled near anchor $k(j)$'s boundary, and $e_k$ is a learned identity embedding for anchor $k$. $W_K$ and $W_V$ are the key and value projections, and $d$ is the embedding width.

The added $\log r_k$ term shrinks an unreliable anchor's share of attention. The token $g_j$ contains signed distances, relative offsets and anchor identity, and nothing from image intensities. A learned per-voxel gate then fuses $a_i$ with the base network's image features to produce the refined target logits.

To stop the module from trusting anchors blindly, training mixes ground-truth anchor shapes with the first stage's own predictions. Anchors are also randomly eroded, shifted and dropped, and $r_k$ is lowered to match each corruption. At test time the module therefore sees anchors of varying quality and has learned how much to rely on them.

The analysis has three parts. We measure how small-organ error depends on anchor error, the "coupling curve." We map which anchors each target attends to, and check this against known anatomical adjacency. We ask whether the gate shifts weight toward geometry on shifted scans and toward image features in-domain.

Baselines

The naive reference point is a 3D U-Net trained on source CT without domain-generalization augmentation and evaluated directly on target scans. It measures the raw gap. AMOS reports 89.04% mean Dice for a 3D U-Net on in-domain CT and a drop of more than 20% on unseen MRI scanners. We found no per-organ number on Lacuna for small organs under that shift, so we will produce it.

The strongest existing method is GIN+IPA from Causality-Inspired Single-Source Domain Generalization. It reports 86.31 Dice on abdominal CT→MRI (T2-SPIR) against a 77.31 baseline. That benchmark covers only large organs. We found no reported result on Lacuna for GIN+IPA on AMOS cross-scanner or small-organ splits. We will rerun it with the authors' augmentation settings on the same backbone, the same patch size and the same epoch budget. Its augmentation strength gets the same hyperparameter search budget as our method, and model selection uses source validation data only. Our first stage is exactly this model, so the comparison isolates the second stage. Beating it means improving mean small-organ Dice on held-out scanners by a margin clearly outside three-seed variance; we set this in advance at 3 Dice points, a threshold of our choosing. Anchor-organ Dice and in-domain CT Dice must not fall.

The closest relational competitor is a CCQ-style module (CCQ). It has the same placement and parameter count as ours, but its keys and values are image features of the anchor regions. It is trained with the same GIN+IPA recipe and run as a second stage on the same first-stage output. This is the critical ablation: it holds "inter-organ attention" fixed and changes only whether the context is geometry or appearance.

We also run Domain Game (Domain Game) as a representation-disentanglement competitor, retrained for multi-organ output, and an EGSDG-style edge-channel input (EGSDG). Both get the same budget.

Three further ablations isolate individual ingredients. The first removes the reliability term ($r_k=1$). The second removes the image branch, giving geometry only. The third gives the module ground-truth anchors at test time; this oracle upper bound shows how much headroom better anchors would provide.

Why not just…?

Why not just use stronger augmentation such as GIN+IPA? It is a real threat, which is why it is our first stage and not merely a comparison. Augmentation teaches invariance to the intensity changes it can simulate. For a pancreas a few voxels wide on an unseen MRI scanner, though, the local evidence can be genuinely missing rather than merely recolored, and AMOS's small-organ failures persist in any case. If adding our second stage on top of GIN+IPA yields no gain beyond seed noise on held-out scanners, augmentation already suffices and the proposal fails.

Why not just use inter-class attention as in CCQ, or a large self-attention backbone? These already let organs inform one another, and a multi-class U-Net sees context implicitly. The difference is what flows through the attention. Image features of the liver change with the scanner; a liver distance field does not. Our matched image-key module tests this directly. If it matches the geometry-key module on shifted scans, the "intensity-free context" hypothesis is refuted and we would report that result.

Why not just apply a shape prior or a post-hoc corrector like ErrorNet? Global priors come from the authors themselves with a caveat: the calibration paper reports that its denoising-autoencoder prior fails on variable multi-organ anatomy, and ErrorNet was shown on vessel trees. We still include a denoising-autoencoder correction of target masks, trained on AMOS ground truth, as a baseline. If it matches our module, the gain comes from shape regularity and not from neighbor conditioning. The anchor-shuffle test below tells these two explanations apart.

Proposed Experiments

All training uses AMOS CT. We test on AMOS MRI, split by scanner model using the scanner information AMOS provides (eight scanner models across two centers); we will confirm per-scan metadata before fixing the splits. We also test on held-out CT scanners as a milder shift. The abdominal CT→MRI setting from Causality-Inspired SDG serves as a sanity check that anchor accuracy matches published levels; it has no small targets, so the main evidence comes from AMOS. MOTS in-domain results check that the module does no harm without shift. Metrics are per-organ Dice and 95th-percentile Hausdorff distance, mean small-organ Dice, relative source-to-target drop, and expected calibration error of the refined targets.

Three checks decide the question.

The first is the geometry-versus-appearance comparison against the matched image-key module on unseen-scanner MRI.

The second is an anchor-shuffle test. At inference we replace each scan's anchor distance fields with those of a different patient, aligned only by bounding box. If accuracy is unchanged, the module is acting as a generic prior and not reading the patient's own anatomy. If accuracy drops sharply, patient-specific neighbor geometry is doing the work.

The third is the coupling curve. We plot target Dice against anchor Dice across target scans, with and without the reliability term. The reliability-weighted module should degrade gracefully when anchors fail rather than spread anchor errors into the small organs. This is the failure mode the research file flags for interaction models that depend on upstream detectors.

The oracle-anchor run then shows whether the remaining errors come from imperfect anchors or from the relational model itself.