Isometric Shape Priors for Test-Time Repair of Segmentations at Unseen Hospitals

A segmentation network trained at one hospital often produces anatomically impossible masks at another. A fetal brain comes out with holes in it, or an abdominal organ is split into islands, because the new scanner's contrast and noise differ from anything seen in training. A common repair is to pass the bad mask through a learned "shape prior", an autoencoder trained only on plausible anatomy, and pull the prediction toward the nearest plausible shape. Every such repair has a knob: how far the mask is allowed to move. In existing shape priors that knob has no fixed meaning. One unit of movement in the prior's internal code can correspond to a tiny boundary nudge for one anatomy and a wholesale change for another. So the knob has to be re-tuned per site, which is impossible when the new site has no labels.

This note proposes to train the shape prior with local isometry regularization, a penalty that forces small steps in the latent code to produce proportionally sized changes in the decoded shape, equally in every direction. The latent code then becomes a ruler for anatomy. A trust-region radius chosen on the source hospital means the same amount of shape change at any target hospital. Because the code is undistorted, the optimization that searches for the best plausible shape is also well-conditioned. The claim to test is concrete: one repair setting, tuned once on source data, transfers to unseen scanners, field strengths and MRI sequences nearly as well as a per-site oracle setting. An unregularized prior is not expected to achieve this.

Core Claim

The shape prior is an implicit decoder $F(x,z)$, a network that takes a spatial coordinate $x$ and a shape code $z \in \mathbb{R}^d$ and returns per-structure occupancy logits. The geometry it induces on the code is the functional pullback metric of Isometric Regularization for Manifolds of Functional Data:

$$H(z) = \mathbb{E}_{x\sim p}\big[\nabla_z F(x,z)\,\nabla_z F(x,z)^\top\big].$$

Here $p$ is a chosen sampling measure over voxel locations. $H$ tells how much the decoded shape, measured as an $L^2(p)$ distance between occupancy fields, changes per unit move of $z$. We penalize deviation from a single global scaled identity:

$$L_{\text{iso}} = \frac{d\,\mathbb{E}_z[\operatorname{tr} H(z)^2]}{\big(\mathbb{E}_z[\operatorname{tr} H(z)]\big)^2} - 1.$$

By Cauchy–Schwarz this is zero exactly when $H(z)=cI$ for one constant $c$ over the sampled codes. The traces are estimated with random probe vectors and Jacobian-vector products, so no voxel-sized Jacobian is ever formed.

When the penalty holds, a first-order Taylor expansion gives $\|F(\cdot,z_1)-F(\cdot,z_2)\|_{L^2(p)} \approx \sqrt{c}\,\|z_1-z_2\|$ for nearby codes. A latent trust region of radius $r$ is then a mask-space trust region of radius $\sqrt{c}\,r$, wherever on the shape manifold the repair happens. At test time, a frozen segmenter's soft output $\hat y$ on a target-site image is encoded to $z_0$. The repair solves

$$z^\star = \arg\min_z \ \ell\big(F(\cdot,z),\hat y\big) + \lambda\|z-z_0\|^2,$$

where $\ell$ is a soft-Dice/cross-entropy fit to the segmenter's probabilities. The output is $F(\cdot,z^\star)$. The prediction is that the source-tuned $\lambda$ transfers across sites only when the prior is isometric. A second prediction is that the condition number of $H$ along test-time repair paths stays near one for the isometric prior, and is large and site-dependent otherwise.

Positioning

Two lines of medical work motivate the gap. ErrorNet trains a VAE shape prior to inject realistic errors and learns an image-space correction network; it reports 71.0 average cross-dataset Dice on retinal vessels. Improved post-hoc probability calibration for out-of-domain MRI segmentation uses a denoising-autoencoder shape prior to score plausibility and rescale confidences of a frozen network. Neither regularizes the prior's geometry. The second explicitly reports that its prior handles only simple round cardiac shapes and only changes probabilities, not delineations. We keep their frozen-segmenter, label-free setting but change what the prior is and use it to move the mask.

Contrastive Domain Disentanglement and Linear-Dependency Regularization instead change how the segmenter is trained so it generalizes. These approaches are complementary: they retrain, while we repair outputs of any fixed model, including clinically validated ones.

The isometry machinery is borrowed. We reuse the JVP-based relaxed distortion loss and functional metric from Heo et al., and latent mixup from Learning Flat Latent Manifolds with VAEs and Isometric Quotient VAEs. Those papers evaluated on pendulums, mocap, ShapeNet, wafers and cells, never on cross-site anatomy. Their limitations also constrain the design. Global flatness of a curved, branching shape space is unrealizable (FMVAE, FlatVI), and our claim needs isometry only locally, around the region that test-time repairs visit. We therefore apply the penalty on encoded training shapes and on their mixups, not over the whole prior. Heo et al. also leave the choice of sampling measure $p$ open. For anatomy this choice is the substance of the method, because uniform sampling makes the ruler measure volume and boundary-concentrated sampling makes it measure surface displacement.

Research Program

We will build a multi-structure implicit shape decoder with an amortized encoder. It is trained on source-site label maps only, rigidly aligned to an atlas so that the code does not spend capacity on patient pose. The loss combines occupancy reconstruction, $L_{\text{iso}}$ on encoded and mixed-up codes, and a flexible prior on codes.

The sampling measure $p$ is a mixture of uniform volume samples and a band within a few voxels of each structure boundary. The mixture weight is studied as a factor.

Analysis has three parts. First, we measure the condition number of $H$ and the variance of $\operatorname{tr}H$ on codes produced by target-site predictions, which the prior never saw. This shows whether source-trained isometry survives the shift. Second, we measure how well $\sqrt{c}\,\|z^\star-z_0\|$ predicts the actual occupancy change. Third, we check whether repair reduces topological errors, without any explicit topology loss, simply because decoded shapes stay near the training anatomy.

Baselines

The naive reference is the frozen segmenter with no repair: a 3D nnU-Net with its default augmentation, the dominant approach on FeTA 2024, and a 3D-UNet on AMOS, where AMOS reports 89.04% mDice on CT and a drop of over 20% on unseen MRI scanners. The same frozen nnU-Net with largest-connected-component post-processing is the cheap topology fix any clinician would try first.

The strongest existing competitors are the unregularized shape priors. The first is a denoising-autoencoder prior used as in Ouyang et al., but applied as a mask projector; for calibration they report cross-sequence ECE of 15.37% on bSSFP→LGE. The second is an ELBO-trained VAE prior as in ErrorNet. Both get the identical implicit decoder architecture, training data, repair objective, optimizer and repair step budget as our method. Each is evaluated twice: once with $\lambda$ tuned on source validation (the fair, deployable comparison) and once with a per-target-site oracle $\lambda$ tuned on target labels (an upper bound no method may use).

The ablation that isolates the key ingredient keeps everything identical and swaps $L_{\text{iso}}$ for spectral normalization on the decoder, a smoothness penalty without isotropy. A second ablation replaces the latent penalty with a mask-space trust region $\lambda\|F(\cdot,z)-F(\cdot,z_0)\|^2_{L^2(p)}$ on the unregularized prior.

Beating the field means our source-tuned repair achieves a smaller gap to its own oracle than every unregularized prior. It must also improve Dice and Euler characteristic difference over the no-repair and connected-component references on held-out sites.

Why not just…?

Why not just use heavier augmentation and connected-component cleanup? FeTA 2024 already shows augmented nnU-Nets losing roughly 0.075 Dice to site and roughly 0.10 to image quality, with topological holes that Dice hides. Connected components remove islands but cannot fill cavities or restore a missing structure. The baseline is run as the no-repair reference, and the Euler characteristic difference metric will show whether repair adds anything beyond it.

Why not just use an ordinary autoencoder prior and tune $\lambda$ carefully? This is the real threat, since prior-based repair is not new. Careful tuning on source data is exactly what fails if the latent scale varies across the shape manifold, and per-site tuning needs target labels. The experiment settles it directly. If an unregularized prior's source-tuned $\lambda$ lands as close to its oracle as ours does, isometry contributes nothing to transfer and we would report that.

Why not just smooth the decoder with spectral normalization? Heo et al. show that Lipschitz-style smoothing blurs the represented manifold and gives nearly 10x worse sparse-data interpolation error than isometric regularization. Smoothness also bounds stretching from above without equalizing it across directions, so it does not make $\lambda$ meaningful. The spectral-normalization ablation tests this on anatomy.

Why not just put the trust region in mask space and skip isometry? That also gives $\lambda$ a physical meaning, and it is a serious alternative. What it does not fix is conditioning. The gradient descent over $z$ still runs through a distorted chart, so equal step budgets may converge unevenly across sites. We run it as an ablation and compare repair quality at matched optimization steps. If it ties us, the honest conclusion is that isometry's contribution reduces to optimization efficiency, and the note's headline claim would be withdrawn.

Proposed Experiments

FeTA 2024 (fetal brain MRI). We train the segmenter and prior on two sites and test on the held-out site, including the 0.55T low-field set. For reference, the best reported low-field result is Dice 0.86, HD95 1.69 and Euler characteristic difference (ED) 6.26, and the challenge-winning topology score is ED 20.9 (FeTA 2024).

AMOS. We train on CT and test on MRI from unseen scanners. This probes a large contrast shift and a 15-structure prior with small organs, where over-regularization is most likely to erase detail.

Cardiac sequence shift. We train on ACDC bSSFP and test on MS-CMRSeg LGE, which is the shift where Ouyang et al. report their calibration numbers. For label-free source-to-target transfer of a repair hyperparameter, no reported result was found on Lacuna, so the oracle-gap metric is new and every baseline will be measured under it.

Metrics. We report Dice, HD95 and ED per site. The headline is the oracle gap: performance at the source-tuned $\lambda$ minus performance at the per-site oracle $\lambda$. Alongside it we report the condition number of $H$ along repair paths and repair quality versus optimization steps.

Decisive checks. The first decisive check is whether the oracle gap of the isometric prior is significantly smaller than that of the denoising, VAE and spectral-norm priors across held-out sites, with seeds and site-level bootstrap. The second is whether the measured condition number on target-site codes stays near one; if isometry learned on source shapes collapses on target codes, the mechanism is falsified regardless of Dice. The third is a sweep of the boundary sampling weight in $p$, testing whether surface-weighted isometry gives better HD95 and ED transfer than volume-weighted isometry, as the choice of ruler predicts.