Edge-Aware Through-Plane Feature Smoothing for Anisotropic Medical Volumes

Most clinical CT and MRI scans are stacks of thin, high-resolution images separated by thick gaps: pixels may be half a millimeter wide while slices sit three to five millimeters apart. Networks that segment such scans either process each slice alone, which throws away the depth context radiologists rely on, or use full 3D convolutions, which treat the large slice gap as if it were a small pixel step. We propose a middle path with almost no added parameters. A shared 2D network processes a short stack of adjacent slices. Inside the network, each slice's intermediate features are blurred with its neighbors' features along the depth axis. The blur width is set in millimeters from the scanner's slice spacing. The blur also switches itself off wherever neighboring slices' features disagree, which is where organ boundaries enter or leave the stack.

Put plainly: within the interior of an organ, adjacent slices should agree, so averaging their internal features removes slice-specific noise. At a boundary they should not agree, so the averaging must stop there. The proposal tests one falsifiable claim. Much of what learned cross-slice attention achieves on anisotropic data comes from this kind of edge-preserving, low-pass consensus between neighboring slices. A fixed operator that knows the physical slice spacing should therefore recover most of that gain, generalize better when slice thickness changes, and give calibrated uncertainty from a single forward pass.

Core Claim and Mechanism

Let $F_z(x)$ be the feature vector at in-plane position $x$ on slice $z$, produced by an encoder stage of a 2D backbone shared across the stack. Let $s$ be the physical slice spacing in millimeters. We replace $F_z$ with

$$\tilde F_z(x) = \frac{\sum_{k=-K}^{K} \exp\!\left(-\frac{(k s)^2}{2\sigma^2}\right)\exp\!\left(-\frac{\|F_z(x)-F_{z+k}(x)\|^2}{\tau}\right) F_{z+k}(x)}{\sum_{k=-K}^{K} \exp\!\left(-\frac{(k s)^2}{2\sigma^2}\right)\exp\!\left(-\frac{\|F_z(x)-F_{z+k}(x)\|^2}{\tau}\right)}$$

In this formula, $k$ indexes neighboring slices up to a window of $K$. The width $\sigma$ is set in millimeters for each encoder stage and grows with that stage's in-plane receptive field. The temperature $\tau$ controls how quickly a feature disagreement shuts the averaging off. The operator is a bilateral filter along depth only, applied to learned features rather than to pixels. Because $\sigma$ is physical rather than counted in slices, a scan with 5 mm slices averages over fewer neighbors than one with 1.5 mm slices.

The second exponential in the formula is a smooth, inverted version of the inter-slice residual $|F_z - F_{z-1}|$ that Inter-slice Context Residual Learning for 3D Medical Image Segmentation (ConResNet) computes. ConResNet uses that residual as an attention signal to emphasize boundaries, which is a purely high-pass operation. We use the same quantity to decide where not to smooth. This settles, in 3D, the tension the file identifies between ConResNet's differencing and the low-pass-plus-high-pass split of Frequency-aware Feature Fusion for Dense Image Prediction (FreqFusion). FreqFusion learns spatially varying low-pass filters for 2D natural images. We need no learned kernel generator, because depth disagreement already marks the boundaries.

Relation to Prior Work

Three papers motivate the mechanism, and they come from 2D and video vision. Curriculum By Smoothing inserts fixed Gaussian blurs after convolutions in 2D CNNs and anneals them during training. Its ablation found that smoothing internal features beats smoothing only the input. Blurs Behave Like Ensembles shows that small average-pool blurs inside 2D networks act like an ensemble, flatten the loss landscape and improve calibration. It also shows that adding temporal smoothing across video frames on CamVid helps further. Neither paper touches medical volumes, where the "frame rate" along depth is far coarser than the in-plane resolution and partial-volume effects dominate. Our operator is the depth analogue of their temporal smoothing, with two additions: kernels scaled to physical spacing, and edge gating.

Four papers constrain the design or serve as competitors. Deep convolutional networks for pancreas segmentation in CT imaging (Roth et al.) applies a 3D Gaussian to stacked 2D output probabilities, raising Dice from 56% to 68%. This proves depth smoothing helps, but only as post-processing; we move it inside the network, where it shapes training. CSAM adds learned semantic, positional and slice attention to U-Net skip connections, plus a low-rank Gaussian uncertainty block. It reports 51.0% Dice on anisotropic prostate cancer segmentation versus 37.0% for 3D nnU-Net. Its pooling produces attention weights rather than smoothing features, and our central test is whether a parameter-free smoother can match it. nnU-Net supplies the preprocessing, training pipeline and 2D and 3D reference models. ConResNet supplies the boundary signal we invert.

Research Program

We will build the operator as a drop-in layer after each encoder stage of the nnU-Net 2D U-Net. At training time the network sees stacks of 8 to 16 consecutive slices, with the slices of a stack forming the batch. The same code will run on CSAM's U-Net++ backbone, so the comparison with CSAM happens on its own architecture.

The analysis has three parts. First, we measure accuracy and boundary quality. Second, we test whether the gains are the "consensus" effect we claim. To do this we record, per voxel, how much the gate suppressed averaging, and relate it to the distance to the ground-truth boundary. The gate should suppress averaging near boundaries and pass it through inside organs. Third, following Park and Kim, we test whether depth smoothing yields single-pass uncertainty. For the uncertainty estimate we compute, per voxel, the gate-weighted variance of neighboring slices' predictions, and compare how well it tracks segmentation error against CSAM's dedicated uncertainty block.

Baselines

The naive reference is the nnU-Net 2D U-Net run slice by slice, with no depth context, under the identical pipeline. It shows how much any depth context is worth on each dataset.

The strongest existing method on anisotropic data is CSAM. On its prostate cancer task CSAM reports 51.0% Dice, against 37.0% for 3D nnU-Net (CSAM). We will retrain CSAM ourselves on the same splits, backbone, stack length and epoch budget. Only if our numbers diverge badly from the reported ones will we quote those instead. Beating CSAM means matching its Dice within the seed-to-seed standard deviation while adding no learned parameters, and clearly beating it on the slice-thickness shift test described below. Matching it only on in-distribution Dice would still support the consensus claim, but it would not make the method preferable.

On pancreas CT, the strongest reported number in the file is ConResNet's 86.06% mean Dice on NIH Pancreas-CT, at roughly an 18.6% increase in compute (ConResNet). ConResNet is a 3D model. We will therefore compare against it as reported and, if code allows, retrained on our splits. We treat reaching within one Dice point at lower compute as a success, not strict superiority.

The post-hoc competitor is Roth et al.'s 3D Gaussian on output probabilities (Roth et al.). We apply it to the naive 2D model and to our model, with its width tuned on validation data.

The native 3D reference is nnU-Net's 3D full-resolution model. For our purposes it stands in for the ordinary full-3D approach, and CSAM reports its prostate cancer result as 37.0% Dice.

Three ablations isolate the key ingredient: the same operator with the gate removed (plain depth Gaussian), with $\sigma$ counted in slices rather than millimeters, and with the fixed kernel replaced by a learned depthwise convolution across slices. Every method gets the same budget of eight hyperparameter configurations on validation folds, and results are averaged over three seeds with nnU-Net's five-fold protocol.

Why not just…?

Why not just smooth the output probabilities in 3D, as Roth et al. did? This is the most serious threat. Roth et al.'s 12-point Dice gain shows that much of the benefit of depth context may be pure output consensus. Curriculum By Smoothing's finding that internal smoothing beats input smoothing is suggestive, but it comes from 2D classification. It does not settle whether internal smoothing beats output smoothing in segmentation. We therefore run both, and combine them, on every dataset. If internal smoothing adds nothing over post-hoc output smoothing, the core claim fails and we will report that.

Why not just stack neighboring slices as input channels, or use a learned depthwise convolution across slices? Learned depth mixing can in principle learn our kernel. It is a real competitor wherever labeled data is plentiful. Our bet is that a fixed, spacing-aware kernel wins when training data is scarce and when slice thickness at test time differs from training. A learned kernel is tied to the slice count it saw, whereas ours is defined in millimeters. The learned-depthwise ablation and a training-data-fraction sweep (10%, 25% and 100% of cases) will show where each wins.

Why not just use CSAM or a 3D nnU-Net? CSAM's own numbers say full 3D convolutions struggle on highly anisotropic MRI. CSAM itself is the method to beat, not something to dismiss. What we offer is a simpler mechanism, an explanation of where the gain comes from, and possibly better robustness to changes in slice thickness. If CSAM wins decisively everywhere, the conclusion is that learned global slice attention captures something beyond local consensus, which is itself informative.

Proposed Experiments

The datasets are the anisotropic MRI tasks used by CSAM (prostate zones, prostate cancer, placenta, and cardiac ACDC), NIH Pancreas-CT, and the MSD prostate and pancreas tasks through nnU-Net's pipeline. We report Dice and 95th-percentile Hausdorff distance. We add a through-plane consistency score: the absolute difference between predicted and ground-truth Dice overlap of consecutive slices' masks, plus the excess number of 3D connected components over the ground truth. For calibration we report voxel-level expected calibration error and the correlation between uncertainty and error, set against CSAM's reported r = 0.791.

Four checks decide the outcome. First, internal smoothing must beat post-hoc output smoothing, and combining them should not erase the difference. Second, the gated operator must beat the ungated one on Hausdorff distance, showing the gate preserves boundaries. The gate's suppression should also concentrate within a few millimeters of true boundaries.

Third, we run a slice-thickness shift test. We train on the finer-spaced cases, or on full stacks, then evaluate on the same volumes with every second or third slice removed. The millimeter-scaled kernel should degrade less than the slice-counted ablation, the learned depthwise convolution, and CSAM. This test most directly exercises what is new here.

Fourth, the single-pass uncertainty estimate should reach error correlation and calibration comparable to CSAM's uncertainty block and to five-pass Monte Carlo dropout, at a fraction of the cost. A curriculum variant that anneals $\sigma$ during training, following Curriculum By Smoothing, is a secondary arm. We keep it only if it helps beyond a fixed $\sigma$.