When a segmentation model trained at one hospital is run on scans from another scanner, its mistakes are not spread evenly. They pile up at organ and lesion boundaries, and in slice-by-slice models they also show up as structures that flicker in and out between neighbouring slices. This proposal tests a cheap fix that leaves the deployed model's weights untouched. We add a small 3D refinement step on top of the model's output. It is a conditional random field (CRF), a probabilistic model that cleans up a label map by trading off each voxel's predicted class against agreement with similar nearby voxels. Our CRF differs from the standard one in three ways. First, "similar" is measured with an image descriptor that is unaffected by any monotone or inverting change in scanner intensity. Second, neighbourhoods are measured in millimetres, so the step respects thick slices. Third, the smoothing term is switched on only where the frozen model is uncertain.
The central bet is about when CRFs help at all. The best existing test of 3D CRFs on medical volumes found no significant gain. That result is easy to explain: on in-domain data the network's per-voxel predictions are already good, so there is little left for a CRF to fix. Under scanner shift, those per-voxel predictions degrade at exactly the places a structure-aware coupling can repair. We predict that the gain from refinement grows with the size of the shift-induced drop, and stays near zero in-domain. If that holds, the result reconciles the negative CRF literature with a practical, regulation-compatible tool for cross-site deployment. If it does not hold, the experiment will say so plainly.
Core Claim
The claim is that a band-limited, anisotropic 3D CRF can recover a meaningful share of cross-scanner losses in boundary accuracy and slice-to-slice consistency for both 2D and 3D frozen segmenters. Its pairwise affinities come from rank-normalized gradient structure, and its coupling strength is gated by the frozen model's predictive entropy. It does this without target-domain data and without updating the base model. The claim is also that this works better than intensity-based CRFs, Gaussian smoothing, and mask-only learned refiners specifically on shifted sites.
Prior work fixes the shape of this claim in three ways. It motivates the problem. Train Once, Deploy Anywhere shows that cross-center errors concentrate at boundaries, and argues that edge structure is more stable across domains than raw intensity. It does so by feeding Sobel and Canny edge maps as extra inputs to a retrained network. We instead use structure only in the output-space coupling of a frozen model. Segmentation Certainty Through Uncertainty shows that a 3D V-Net under multifactor shift over-segments along boundaries, and that dropout-based uncertainty can locate these errors. Its fix only removes voxels. A CRF can also add voxels, which lets it close holes and reconnect broken structures.
Prior work also constrains the design. Conditional Random Fields as Recurrent Neural Networks for 3D Medical Imaging Segmentation built an n-dimensional, end-to-end CRF with intensity-based Gaussian kernels on a V-Net. It saw non-significant changes on PROMISE 2012 (Dice 0.767 to 0.780) and BraTS 2015 (0.735 to 0.738), and blamed soft medical edges. We take this as a real warning, and our design responds to it in two ways: we drop raw-intensity kernels, and we evaluate where the per-voxel predictions are actually damaged. Semi-Supervised Semantic Segmentation via Marginal Contextual Information shows that fixed-window neighbourhood aggregation raises mIoU but lowers Boundary IoU. Any refinement we propose must therefore be judged on boundary metrics, not Dice alone.
Finally, prior work supplies the method. We reuse the permutohedral-lattice mean-field machinery from the 3D CRF-as-RNN paper. We also borrow from Improved post-hoc probability calibration for out-of-domain MRI segmentation the idea that a frozen model can be corrected after the fact by a spatially varying signal built from local susceptibility to perturbation.
Mechanism
Let $p_i$ be the frozen model's class probabilities at voxel $i$, and let $y$ be the refined labelling. We minimise
$$E(y) = -\sum_i \log p_i(y_i) + \sum_{i<j} g_i g_j \, \mu(y_i, y_j)\, k(f_i, f_j, \Delta_{ij}).$$
Each symbol is defined as follows:
- $g_i \in [0,1]$ is a monotone function of the frozen model's predictive entropy at $i$. Confident voxels therefore barely move.
- $\mu$ is a learned class-compatibility matrix.
- $\Delta_{ij}$ is the physical displacement between voxels in millimetres. It has separate in-plane and through-plane bandwidths, so a 5 mm slice gap is not treated like a 0.5 mm pixel step.
- $f_i$ is a structural descriptor, described next.
The descriptor $f_i$ is built in two steps. First, intensities are rank-normalized within each scan's body mask. Then we take multi-scale gradient magnitudes of the rank image. Rank normalization cancels any monotone remapping of intensities, which covers the Bézier-style shifts that Train Once, Deploy Anywhere uses to simulate scanner change. Using gradient magnitude rather than signed gradient also cancels contrast inversion, which is a large part of the T1-versus-T2 and CT-versus-MRI gap.
The kernel $k$ is a sum of a spatial Gaussian and a Gaussian in $f$. Inference is run only inside a narrow band around the frozen model's uncertain region. This sidesteps the quadratic cost that limits dense 3D CRFs.
The whole module has a few dozen parameters: bandwidths, compatibilities, and the gate's slope and offset. They are fit on held-out source-domain volumes under source-only intensity and artifact augmentation, then frozen. A model this small has little capacity to overfit the source scanner, and that is deliberate.
Research Program
The work proceeds in three steps.
The first step builds the refiner as a CUDA band-limited mean-field layer, extending the permutohedral filtering of the 3D CRF-as-RNN work. It is attached to three frozen base models:
- a 2D slice U-Net, which is the regime where volumetric context loss is worst, as Deep convolutional networks for pancreas segmentation in CT imaging documents;
- a native 3D U-Net of the kind benchmarked in AMOS;
- an edge-guided single-source domain generalization model following Train Once, Deploy Anywhere. This tests whether output-space refinement adds anything once structure has already been used at the input.
The second step measures, per test site, the base model's drop from in-domain performance. It then regresses the refiner's gain on that drop. This is the analysis that directly tests the central bet.
The third step characterises failure. We stratify gains by boundary contrast, structure size, and slice thickness, because thin, low-contrast structures are where the edge-guided approach is known to degrade.
Baselines
The naive reference is the frozen base model's output with argmax and no refinement. Every gain is reported relative to it on the same test volumes.
The classical post-processing baselines are two. The first is isotropic 3D Gaussian probability smoothing as in Deep convolutional networks for pancreas segmentation in CT imaging, which reports raw 2D ConvNet Dice of 56% rising to 68% ± 10% after smoothing on NIH pancreas CT. The second is 3D connected-component filtering with morphological closing. Both get their kernel sizes tuned on the same source validation set and the same search budget as our parameters. Beating them means equal or better Dice together with clearly better 95th-percentile Hausdorff distance and surface Dice on shifted sites, since smoothing is expected to help Dice while blurring edges.
The intensity-kernel dense 3D CRF of Conditional Random Fields as Recurrent Neural Networks for 3D Medical Imaging Segmentation is the closest existing probabilistic refiner. Its reported numbers are the in-domain reference: PROMISE 2012 Dice 0.780 against 0.767 for the V-Net, and BraTS 2015 whole-tumor Dice 0.738 against 0.735, neither significant. We run it post hoc on the identical frozen base models, with its kernel parameters fit on the same source data.
The strongest learned competitor is a mask-only 3D refiner. It is a 3D extension of ErrorNet, which learns an additive correction map from synthetically corrupted masks. ErrorNet reports 71.0 average cross-domain Dice on retinal vessels, but it is 2D only, and no reported result for a 3D version was found on Lacuna. We train it on the same source volumes, with the same corruption types and GPU-hour budget, in two variants: one seeing only the probability volume and one also seeing the raw image. Beating it means a statistically significant improvement in surface Dice on shifted sites with comparable in-domain performance.
For deployment context we also report the strongest domain-generalization numbers in the file. Train Once, Deploy Anywhere reports 64.78% Dice in BraTS'19 cross-sequence generalization. On multi-source prostate, no reported per-center result was found on Lacuna.
Ablations isolate each ingredient on the same frozen bases:
- an intensity affinity in place of the rank-gradient descriptor;
- removing the entropy gate, so coupling is uniform;
- an isotropic voxel-space kernel in place of millimetre-anisotropic bandwidths;
- 2D per-slice refinement in place of 3D;
- a full volume in place of band-limited inference.
Why not just…?
Why not just smooth the probabilities in 3D, as Roth et al. did? Gaussian smoothing does fix slice flicker, and its 56%-to-68% gain on pancreas is real. It is the baseline most likely to match us on Dice. However, it smooths straight across true boundaries, and Semi-Supervised Semantic Segmentation via Marginal Contextual Information shows that fixed-window aggregation lowers Boundary IoU. Our comparison reports surface Dice and Hausdorff distance alongside Dice so that this trade-off is visible. If smoothing ties us on boundary metrics on shifted sites, the structural affinity is not earning its cost, and we will say so.
Why not just train a 3D U-Net with aggressive intensity augmentation? This is a serious alternative. If good augmentation closed the cross-scanner gap, there would be nothing left to refine. The evidence says it does not close it yet. AMOS reports 89.04% mDice for 3D U-Net in-domain on CT, yet drops above 20% mDice on unseen MRI scanners. The 3D V-Net in Segmentation Certainty Through Uncertainty reaches 99.1% in-domain accuracy and becomes unusable under shift. More importantly, we do not compete with such a model; we stack on it. One of our frozen base models is exactly an augmented, edge-guided network. The regression of gain against shift drop then tells us how much residual error augmentation leaves for output-space refinement.
Why not just learn a mask-to-mask denoiser instead of a CRF? This is the alternative most likely to make the idea unnecessary. A 3D network that cleans up masks can learn shape priors that a CRF cannot represent. Our argument for the CRF is that it ties the correction to image structure in a way that is invariant by construction, and that it has too few parameters to memorise source-scanner quirks. The learned refiner could instead overfit the source appearance, or ignore the image and "correct" toward an average shape. The image-plus-mask variant of the 3D ErrorNet baseline tests this directly. If it matches us on shifted-site surface Dice, the honest conclusion is that learned refinement wins, and the CRF's value reduces to interpretability and low parameter count.
Why not just adapt to the target site with unsupervised domain adaptation? Methods such as SDC-UDA reach 84.8 mean Dice on CrossMoDA, but they require unlabelled target volumes during training and retrain the model. As Adapt the Data, Not the Model notes, frozen deployed weights often rule this out. We report the SDC-UDA number as a ceiling that uses target data, not as a competitor in the same setting.
Proposed Experiments
The main shift testbeds are two:
- AMOS, training on CT and one MRI scanner and testing on held-out MRI scanners, to probe the reported >20% collapse;
- the six-site multi-source prostate collection, training on one site and testing on the other five.
These test the bidirectional, structure-aware claim on abdominal organs and prostate. In-domain controls are PROMISE 2012 and BraTS 2015, used to reproduce the null CRF result. MS-CMRSeg, trained on bSSFP and tested on LGE, covers thick-slice cardiac stacks where anisotropy matters. NIH pancreas CT covers a non-star-shaped, low-contrast organ that defeats Elastic Boundary Projection.
Metrics are Dice, surface Dice at a tolerance set from slice thickness, 95th-percentile Hausdorff distance, and Boundary IoU. Two further metrics are a slice-consistency score, defined as the fraction of slices where a structure appears and disappears against its 3D connected extent, and a count of spurious connected components. Expected calibration error after refinement is also reported. All comparisons use paired Wilcoxon tests across volumes, grouped by site.
The decisive check is the per-site regression of refinement gain on base-model shift drop. The bet is supported if the slope is positive and significant, and in-domain gains stay within Monteiro et al.'s non-significant range. A secondary decisive check is the intensity-affinity ablation on the shifted sites. If swapping the rank-gradient descriptor for raw intensity removes most of the gain, invariance is the working ingredient. If it does not, the gain is coming from the gate and the anisotropic geometry, and the paper is reframed around those.
References
- Train Once, Deploy Anywhere: Edge-Guided Single-source Domain Generalization for Medical Image Segmentation
- Segmentation Certainty Through Uncertainty: Uncertainty-Refined Binary Volumetric Segmentation Under Multifactor Domain Shift
- Conditional Random Fields as Recurrent Neural Networks for 3D Medical Imaging Segmentation
- Semi-Supervised Semantic Segmentation via Marginal Contextual Information
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- Deep convolutional networks for pancreas segmentation in CT imaging
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- ErrorNet: Learning Error Representations from Limited Data to Improve Vascular Segmentation
- SDC-UDA: Volumetric Unsupervised Domain Adaptation Framework for Slice-Direction Continuous Cross-Modality Medical Image Segmentation
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Elastic Boundary Projection for 3D Medical Image Segmentation