Scanner-Invariant Anatomy Graphs for Catching and Repairing Segmentation Failures

When a multi-organ segmentation network moves to a new scanner, it rarely fails everywhere at once. It might lose the pancreas, swell the duodenum, or merge the gallbladder into the liver, while the large organs still look fine. The images change with the scanner, but the patient's anatomy does not: how big the spleen is given the size of the liver and the position of the kidneys reflects the body, not the hardware. We propose to learn this anatomy as a sparse Gaussian graphical model. That is a statistical model in which each organ measurement (volume, centroid, extent) is predicted from a small set of directly related measurements. The model is encoded in the inverse covariance, or "precision," matrix of these measurements. A zero entry in that matrix means two measurements are unrelated once all the others are known. We fit the model on source-domain ground-truth labels. On a new scanner, we score the network's own predictions by how far each organ deviates from what its anatomical neighbours imply, which needs no target labels. The same score then drives a forward-only repair step: for each organ, pick among candidate segmentations the one most consistent with the rest of the body.

The reason to use the precision matrix rather than plain correlations is that the error signal becomes local. Under marginal correlations, a wrong liver makes every liver-correlated organ look anomalous. Under conditional residuals, the anomaly concentrates on the organ that is actually wrong. This differs from the rigid shape templates that current test-time adaptation uses and that the MedSeg-TTA benchmark shows can cause negative transfer on atypical patients (A Large Scale Benchmark for Test Time Adaptation Methods in Medical Image Segmentation). A conditional model accepts an unusual body as long as its organs are mutually consistent.

Core Claim

The claim has two parts. First, a precision matrix over organ-level anatomical measurements, estimated from labels, transfers across scanners and modalities far better than any image- or feature-level statistic. Second, per-organ conditional residuals computed from the network's predictions predict per-organ segmentation failure under domain shift better than marginal plausibility scores or predictive entropy. Used as a selection and early-stopping signal, they add Dice and boundary gains on top of the strongest simple adaptation recipe.

Write $x \in \mathbb{R}^p$ for the vector of organ measurements extracted from a segmentation, $\mu$ for their source mean, and $\Theta$ for the estimated sparse precision matrix. The conditional standardized residual of measurement $i$ is then

$$z_i(x) = \frac{\left(\Theta (x-\mu)\right)_i}{\sqrt{\Theta_{ii}}},$$

which equals the deviation of $x_i$ from its Gaussian conditional mean given all other measurements, divided by the conditional standard deviation $1/\sqrt{\Theta_{ii}}$. An organ's anomaly score is the sum of $z_i^2$ over its own measurements. Sparsity in $\Theta$ means each residual depends only on a few anatomical neighbours, which is what makes the flag local.

The idea's role in each direction is uneven by design. Sparse precision estimation supplies the core mechanism. Medical domain shift is the regime in which it is tested and the source of the invariance argument.

Relation to Prior Work

Several papers connect precision matrices to neuroimaging, but none applies them to domain shift.

- Learning Brain Connectivity Sub-networks by Group-Constrained Sparse Inverse Covariance Estimation shares edge topology across subjects at a single site. It provides motivating evidence that conditional structure beats raw correlation: 81.82% accuracy versus 58.18% for simple partial correlation.

All three model brain signals that are themselves scanner-dependent. We instead model anatomy extracted from labels, which is chosen precisely because it should not depend on the scanner.

On the domain-shift side, existing methods align image or feature statistics rather than relational anatomy:

Our method touches no features at all, so it is compatible with any of these.

The closest prior work in spirit is the post-hoc calibration method of Improved post-hoc probability calibration for out-of-domain MRI segmentation. It uses a denoising-autoencoder shape prior for a single regular structure, the heart. We replace a holistic shape prior with an explicit conditional graph over many organs, which can say which organ is wrong.

Two papers constrain the method's design.

Research Program

Building the anatomy graph

We use AMOS (AMOS benchmark), which has 500 CT and 100 MRI scans with 15 labelled organs, drawn from two hospitals and eight scanner models.

  • Measurements. From each ground-truth label map we extract, per organ, log-volume, centroid relative to a body bounding frame, and principal extents. This gives roughly 75 measurements.
  • Sex-specific organs. Prostate and uterus are handled by conditioning on sex or by fitting the model only on shared organs.
  • Estimators. We fit $\Theta$ with three methods. The first is the graphical lasso on rank-based correlations. The second is nodewise regression followed by an unpenalized refit on the selected neighbourhoods, which removes the shrinkage bias that the inconsistency result warns about. The third is a dense, shrinkage-regularized inverse covariance.
  • Why compare sparse and dense. With 500 CT scans and about 75 measurements the sample is not severely high-dimensional. Whether sparsity matters beyond conditioning is therefore an empirical question we test, not an assumption. Sparsity should matter most when the source is the 100-scan MRI set.

Testing the invariance premise

Before any adaptation, we fit $\Theta$ separately on CT labels, on MRI labels, and on each scanner group's labels. We compare edge sets and the conditional residual distributions of held-out ground truth across these groups. This uses target labels only for evaluation. If the anatomy graph differs substantially across scanners, the premise fails and we report that.

Label-free failure detection

We train a source segmenter: a 3D-UNet with nnU-Net-style heavy augmentation, as in M3DA. On target scans, we compute per-organ anomaly scores from its predictions and measure how well they rank organs by true Dice.

Forward-only repair

For each target scan we gather K candidate predictions, which require no backpropagation through the deployed model. They come from:

  • test-time augmentations,
  • dropout samples,
  • successive iterates of entropy-minimization adaptation.

We then run iterated conditional selection. Starting from the base prediction, each organ in turn takes the candidate mask that minimizes its conditional anomaly given the current choices for all other organs. We repeat until the choices stop changing.

The total anomaly also serves as a stopping rule for entropy-minimization adaptation. Adaptation halts when anatomical consistency starts to worsen. This targets the drift toward over-segmentation that entropy minimization is known to produce (On-the-Fly Test-time Adaptation reports TENT at Dice 61.20 against a 61.95 unadapted baseline on CHASE to HRF).

Baselines

Unadapted source model. The naive reference point is the source 3D-UNet with heavy augmentation and no adaptation, evaluated on each target split. AMOS reports a drop of more than 20% in mean Dice on unseen MRI scanners for standard training. It also reports a 3D-UNet in-domain CT mean Dice of 89.04%, which bounds the oracle side.

Augmentation plus entropy minimization. The strongest existing method is this combination from M3DA. It closed 62% of the source-to-oracle gap averaged over eight shifts and outperformed many dedicated adaptation methods. On the AMOS cross-modality task (reported in the file as MRI to CT), M3DA reports a no-adaptation Dice of 0.032 and a best-method Dice of 0.388. We reproduce this baseline with M3DA's protocol on identical splits, backbone, and adaptation budget. Our repair step operates on top of it, so any gain is additive.

Rigid shape prior. We train the denoising-autoencoder shape prior of Improved post-hoc probability calibration on multi-organ source labels and use it for the same candidate selection. No reported result was found on Lacuna for this method on AMOS.

Ablations. Two ablations isolate the key ingredient.

  • Marginal ablation: replace $\Theta$ with a diagonal matrix, so each measurement is scored by its marginal z-score. This tests whether conditioning matters.
  • Dense ablation: use the dense shrinkage inverse. This tests whether sparsity matters.

Uncertainty baseline. For failure detection we also compare against mean predictive entropy per organ and against disagreement among the K candidates.

All methods see the same candidate pool, the same source labels, and the same tuning budget, with hyperparameters set by source-domain cross-validation only.

What counts as beating them. We beat the baselines if two things hold on cross-scanner splits. First, failure-detection ranking (Spearman correlation with per-organ Dice, and AUROC for identifying organs with Dice below 0.5) is clearly higher than both the marginal ablation and entropy. Second, repair improves Dice and the 95th-percentile Hausdorff distance over augmentation plus entropy minimization, with gains concentrated on small organs.

Why not just…?

Why not just use heavy augmentation with entropy minimization? It is the real threat: M3DA shows it beats most dedicated methods and closes 96% of the gap on standard- to low-dose CT. Our method does not compete with it but sits on top of it, using the anatomy graph to choose among and stop its iterates. If the repair gain over augmentation plus entropy minimization is within seed noise on cross-scanner AMOS splits, the repair half of the proposal fails. In that case only the detection result would remain.

Why not just score each organ's volume against its source distribution? That is the marginal ablation. It catches gross failures but cannot tell a wrong liver from a wrong neighbour. A large body can also make many organs marginally extreme without any error. The decisive comparison is per-organ AUROC on scans with exactly one badly segmented organ. If marginal scores match conditional ones there, the precision matrix earns nothing.

Why not just use a learned shape prior or a dense covariance? MedSeg-TTA documents negative transfer from rigid anatomical priors on atypical patients. We will report results separately on the decile of target scans whose ground-truth anatomy is most anomalous under the source model, where a rigid prior should hurt most. The dense inverse covariance is a fair alternative when source data are plentiful, and we expect it to be competitive on CT-trained models. Sparsity is predicted to matter for the 100-scan MRI source and for interpretable failure localization. If it does not, we will say so.

Why not just harmonize features (whitening, moment mixing, mean shifting)? These methods change what the network sees. Ours checks what it outputs against invariant anatomy, so the two are complementary. We include an Extended MixStyle–trained source model as an extra backbone to show the repair gain persists on a model that has already been feature-harmonized.

Proposed Experiments

Splits. The primary setting is leave-one-scanner-out within AMOS for both CT and MRI. Here predictions are partially correct and organ measurements carry signal. The cross-modality CT-to-MRI and MRI-to-CT tasks serve as a stress test. At M3DA's 0.032 unadapted Dice, the extracted measurements may be meaningless, and we expect repair to help only once augmentation plus entropy minimization has lifted predictions to a usable level.

Metrics. We report:

  • per-organ Dice and 95th-percentile Hausdorff distance, with small organs (pancreas, adrenals, duodenum, gallbladder, esophagus) reported separately;
  • Spearman correlation and AUROC for failure detection;
  • for the estimator comparison, edge stability across source subsamples and across scanners.

Decisive checks. Three outcomes decide the proposal:

  1. Whether the CT-fit and MRI-fit anatomy graphs agree on held-out ground truth. This is the invariance premise.
  2. Whether conditional residuals beat both marginal residuals and entropy at finding the failing organ.
  3. Whether conditional selection adds boundary-level gains over augmentation plus entropy minimization without degrading the most atypical decile of patients.

We additionally compare the graphical lasso to the refit nodewise estimator on hub organs. This tests whether the shrinkage failure predicted by the inconsistency result shows up in anatomy as it does in gene networks.