Training 3D Segmenters to Respect How Slices Are Acquired

A CT or MRI volume is not a stack of perfect cross-sections. Each slice is the scanner's average of tissue over a slab of some thickness, sampled at some spacing, and both numbers vary from hospital to hospital and protocol to protocol. A 3D segmentation network trained on thin-slice scans from one site learns features measured in voxels. When it meets thick-slice scans from another site, its view along the depth axis effectively collapses toward 2D. It loses the cross-slice continuity that the volumetric literature shows matters: Roth et al. gained 12 Dice points on pancreas CT simply by smoothing 2D predictions across slices (Deep convolutional networks for pancreas segmentation in CT imaging). Small organs that span few slices suffer most. They are exactly the organs on which native 3D models already struggle on AMOS (AMOS).

We propose to write down the physics of slice formation as a forward operator and train against it, in two ways. First, a consistency loss requires that segmenting a thick-slice version of a scan gives the same answer as physically "acquiring" the thin-slice segmentation at that thickness. This target is a soft, partial-volume label, not a sharp mask copied onto blurrier input. Second, a multi-scale dilated convolution block sets its dilation rates (the gaps between kernel taps that widen a filter's view without adding weights) in millimetres rather than voxels, so each branch sees a fixed physical neighbourhood whatever the slice spacing. The claim is narrow and testable. A meaningful share of cross-site failure in 3D segmentation comes from mismatched sampling geometry rather than contrast. Modelling that geometry explicitly should beat the standard remedies of resampling and degradation augmentation, especially on small organs.

Core Claim

Let $x$ be a thin-slice volume with label $y$. Let $A_{s,\Delta}$ be the acquisition operator that convolves along the depth axis with a slice-sensitivity profile of width $s$ and resamples at spacing $\Delta$. Let $f$ be the segmenter that outputs per-voxel class probabilities. Applied to a one-hot label, $A_{s,\Delta} y$ gives the fraction of each thick voxel occupied by each organ, which is the partial-volume effect. We train with

$$\mathcal{L} = \mathcal{L}_{\text{seg}}\big(f(x), y\big) + \lambda\, \mathbb{E}_{(s,\Delta)\sim P}\, D\big(f(A_{s,\Delta} x),\, A_{s,\Delta} y\big),$$

where $D$ is a soft Dice plus cross-entropy on probabilities, $P$ is a distribution over clinically plausible thickness and spacing pairs, and $\lambda$ weights the term. On unlabelled thin-slice scans, $A_{s,\Delta}\, \mathrm{sg}[f(x)]$ replaces $A_{s,\Delta} y$, where $\mathrm{sg}$ denotes stop-gradient. This is an equivariance constraint: the network should commute with the scanner's sampling operator.

The constraint differs from ordinary thick-slice augmentation in what the network is asked to output. Augmentation pairs blurred input with a sharp mask the scan no longer supports. That forces the network to hallucinate sub-slice boundaries, and plausibly teaches it to be overconfident at organ ends.

Relation to Prior Work

The closest work is Physics-Informed Brain MRI Segmentation. It feeds MR acquisition parameters (repetition, echo and inversion times, flip angle) into a branch of a 3D U-Net and synthesises training volumes from signal equations. This cut volumetric disagreement across a scanner upgrade on SABRE to 2%, against 3% and 5% for the comparison pipelines. That paper models contrast physics and conditions on it. We model geometric sampling physics and impose it as a consistency constraint, so the two are complementary. Its scalar-conditioning design is also one of our baselines.

Other physics-informed imaging work supplies the method template but not the target. Biomechanics-informed Neural Networks for Myocardial Motion Tracking in MRI regularises 2D registration with a learned biomechanical prior and improved unseen-ACDC Dice from 0.745 to 0.783. Its authors name the 2D restriction as a limitation, which motivates our insistence on a depth-axis model. Non-rigid Medical Image Registration using Physics-informed Neural Networks shows that a PDE loss improves unseen-subject generalisation. Inverse Evolution Layers shows that physical operators placed before the loss can shape segmentation outputs. Both encourage the "physics as a loss term" route, but neither concerns acquisition sampling.

On the robustness side, Frequency Domain Adversarial Training for Robust Volumetric Medical Segmentation uses a consistency loss under 3D frequency perturbations. It shows that disrupting cross-slice structure drops UNETR on Synapse from 74.3% to 32.2% Dice. Its perturbations are adversarial rather than physically realisable; ours are restricted to the operator a real scanner applies. Post-hoc fixes act only on the output. These include MC-dropout thresholding (Segmentation Certainty Through Uncertainty) and shape-prior temperature maps (Improved post-hoc probability calibration for out-of-domain MRI segmentation). We treat them as add-on baselines rather than competitors to the representation.

On architecture, the most compact multi-scale designs are 2D. PSConv packs cyclic dilation rates into one layer with no extra parameters. SCSC runs parallel depth-wise kernels of different sizes. Cascaded Dilated Dense Network uses dilation rates of 1, 2 and 4 next to physical data-consistency layers, and lists 3D context as open. The 3D dilated designs fix dilation in voxels and never test under cross-site shift (Volumetric Densely Dilated Spatial Pooling ConvNets, Group Shift Pointwise Convolution). Our modest architectural contribution is to make dilation a function of physical spacing.

Research Program

We build three components on a plain 3D U-Net. We use a convolutional backbone because AMOS reports 3D-UNet ahead of Swin-UNetr on CT, at 89.04% against 86.32% mDice.

The first component is a differentiable slice-formation operator. It uses Gaussian and rectangular slice-sensitivity profiles, with profile shape treated as a nuisance variable and randomised, because the true profile differs between CT reconstruction kernels and MRI slice-select pulses.

The second is the partial-volume consistency loss above.

The third is a spacing-aware multi-scale block. It has parallel 3D branches with target physical radii. In-plane and depth dilations are computed per scan from the voxel spacing, and weights are shared across spacings. When the depth spacing exceeds a branch's radius, that branch degrades gracefully to in-plane dilation and does not sample across slabs it cannot resolve.

The analysis piece matters as much as the method. We decompose cross-scanner degradation into a sampling-geometry part and a residual, appearance-driven part. To do this, we measure how much of a source-to-target drop survives when source scans are re-acquired through $A_{s,\Delta}$ to match the target's thickness and spacing.

Baselines

The naive reference is a 3D U-Net trained on the source site at native spacing and applied directly to target scans. It shows how large the geometric shift is at all.

The strongest practical competitor is the same network trained and tested after resampling every volume to a common target spacing, with the full standard intensity and spatial augmentation pipeline. This is the default remedy for anisotropy in current practice. It is strong because interpolation already equalises voxel units. Its likely weakness is that interpolated thick slices remain blurred along depth, so the input distribution still differs.

The second strong competitor is thick-slice simulation augmentation with hard labels. It uses the same operator $A_{s,\Delta}$ and the same distribution $P$, but the loss uses the original sharp mask. It is the ablation that isolates our key ingredient, the partial-volume target.

We also run spacing-as-input conditioning in the style of Physics-Informed Brain MRI Segmentation, with thickness and spacing fed through a small fully connected branch. Further baselines are a fixed-voxel-dilation multi-scale block, to isolate millimetre dilation, and Frequency Domain Adversarial Training as a generic volumetric-robustness method. Post-hoc 3D Gaussian smoothing as in Roth et al. and MC-dropout refinement are applied on top of every model.

All arms share the backbone, patch size, training epochs, optimiser, data and augmentation pipeline, and the same number of hyperparameter trials. Trials tune $\lambda$ for ours and augmentation probability for the augmentation arm, selected on a source-only validation split so that no target labels leak. Multi-scale arms are matched in FLOPs.

The reference numbers available are AMOS's 89.04% CT mDice for 3D-UNet and its report of an over-20% mDice collapse on unseen MRI scanners. On Lacuna I found no reported result for any method under controlled slice-thickness shift, and none for AMOS cross-scanner CT with spacing stratification. Beating the baselines means a statistically reliable gain over resampling plus hard-label augmentation, paired across target scans, in small-organ Dice (adrenal glands, duodenum) and 95th-percentile surface distance on real target scanners, with in-domain mDice within one point of the naive model.

Why not just…?

Why not just resample everything to isotropic spacing? Resampling fixes the voxel unit but cannot restore depth information a thick slice never recorded. The network still sees a blurrier input than it trained on. We expect this baseline to close much of the gap for large organs like liver and spleen and little of it for small ones. If it matches us on small organs at matched compute, the architectural half of the proposal is unnecessary and we will say so.

Why not just augment with simulated thick slices? This is the real threat. It shares our operator and differs only in the target. Our argument is that sharp masks on partial-volume input reward invented boundaries and overconfidence at organ ends. We test this directly in the matched-exposure ablation, scoring both Dice and calibration near the organ's first and last slices. If the hard-label arm ties on both, the contribution reduces to the analysis and the spacing-aware block.

Why not just smooth the output across slices, or filter it with uncertainty? Roth et al.'s 56%-to-68% gain shows that output smoothing helps when predictions are discontinuous. It cannot recover a structure the encoder never detected, and Segmentation Certainty Through Uncertainty removes false positives but cannot add missed voxels. We apply both on top of every arm. The proposal stands only if its gain survives that common post-processing.

Why not just condition the network on slice thickness? Conditioning tells the network the spacing but gives it no reason to produce consistent answers across spacings. It also extrapolates poorly to thickness values outside the source site's range. We test extrapolation explicitly by training on a narrow thickness range and testing beyond it.

Proposed Experiments

The controlled experiment trains on the thinnest-slice AMOS CT scans and tests on the held-out scans of those same patients re-acquired through $A_{s,\Delta}$ over a sweep of thicknesses. We use a profile shape withheld from training to avoid testing on our own simulator. This yields dose-response curves of per-organ Dice against thickness mismatch for every arm.

The real-shift experiment uses AMOS's eight scanners. We train on one centre's scanners, test on the others, and stratify results by native spacing difference. We then transfer AMOS-trained CT models to Synapse and AbdomenCT-1K on overlapping organs.

The MRI experiment applies the decomposition analysis to AMOS's unseen-scanner MRI setting. It asks what fraction of the reported over-20% drop is geometric. A small fraction would itself be a useful negative finding, pointing robustness research toward contrast models such as the physics branch of Physics-Informed Brain MRI Segmentation.

Metrics are per-organ Dice, 95th-percentile surface distance, organ volume error (which partial volume biases directly), and expected calibration error restricted to boundary slices.

Three checks are decisive. The partial-volume target must beat hard-label augmentation at identical exposure. Millimetre dilation must beat resampling at matched FLOPs on small organs. Gains on real scanners must track the measured spacing mismatch. If gains appear in simulation but not on real scanners, our slice profile is wrong. We will test that by fitting profile width per scanner from edge-spread measurements on air–tissue boundaries and rerunning.