A 3D medical scan is usually too large to pass through a segmentation network in one go. Standard practice is to cut it into overlapping cubes, predict each cube, and average the overlapping predictions. This is called sliding-window inference. Heavy overlap consistently helps. On whole-body PET, moving from a single non-overlapping pass to 75% overlap with center-weighted blending raised Dice from 81.02% to 83.83% (Hadlich et al.). The cost is steep: at 75% overlap along three axes, an interior voxel is recomputed up to 64 times. Everyone treats this as the price of robustness, and nobody has asked what the price actually buys.
This note argues that the overlap gain is two separable effects hiding under one knob. The first is avoiding predictions made near a cube's edge, where the network sees truncated context. The second is a genuine ensemble: each window gives a slightly different view of the same voxel. The first can be obtained exactly by computing each voxel's features once and reusing them across neighbouring windows. The second can be obtained cheaply by re-running only the small, low-resolution deep layers at a few deliberately shifted sampling grids. We further hypothesize that the second effect matters most under scanner and vendor shift, where network features are off-distribution. We also expect the disagreement among the cheap shifted branches to serve as a free map of where the model is wrong. If this holds, hospitals get overlap-level accuracy and a usable uncertainty signal on unseen scanners at a small fraction of the compute.
Core Claim
One structural fact makes the decomposition testable, and it appears to have been overlooked. Consider typical configurations, such as the DynUNet-style inferer used by Hadlich et al.. There the cube size and the step between cubes are multiples of the network's total downsampling factor. In that case every overlapping window lands on the same grid of strided convolutions and pooling. The effect behind Giusti et al.'s MaxPoolingFragment layer, where a one-pixel shift changes which input wins the pooling, is therefore never exercised by standard overlap.
Standard overlap thus ensembles only over context truncation. Each window sees a given voxel with a different amount of surrounding anatomy cut off by zero padding. It never ensembles over sampling phase, meaning the offset of the voxel relative to the network's stride grid. Our claim has three parts:
- The in-domain overlap gain is mostly context repair. Exact feature reuse recovers it at roughly single-pass cost.
- The additional robustness under domain shift comes from ensembling, and a phase ensemble branched only at deep layers delivers it more cheaply than spatial overlap.
- Per-voxel disagreement across phase branches predicts out-of-domain errors about as well as multi-pass augmentation.
A competing hypothesis is equally live. Networks trained on truncated patches may need truncated context at test time. Moturu and Joshi showed that patch-trained models break when deployed across whole volumes, missing 58 of 120 tumor slices. If full-context inference is worse than the truncated-window ensemble, the overlap gain is a train–test context-matching effect. The program is designed to tell these apart.
Research Program
The first component is halo-cached inference. We tile the volume and compute each encoder level once per tile. Each tile carries a margin, or halo, sized to that level's receptive field. Features in the overlapping margins are cached and reused rather than recomputed. The result is exactly equivalent to whole-volume inference while keeping peak memory bounded by tile size. This is the volumetric analogue of the fully convolutional reformulation of Oquab et al.. It also sidesteps the memory blow-up that Giusti et al. incur by holding all $k^2$ fragment maps at once. The trade-off here is a fixed halo width rather than memory for every offset simultaneously.
The second component is deep-branch phase ensembling. Take a U-Net whose stage $s$ halves resolution along each axis while doubling channels. For a 3D convolution, stage $s$ then costs about half of stage $s-1$: there are 8× fewer voxels but 4× more multiply-accumulates per voxel. Because of this halving, recomputing all stages from $s$ onward costs roughly a $2^{-(s-1)}$ fraction of the full network.
We produce $P$ phase-shifted predictions as follows. Take the cached full-resolution features entering stage $s$. Apply the stride-2 downsampling at offsets other than zero. Run only stages $s$ and deeper, plus the matching decoder path, for each offset. Average the results. The added cost is about $P \cdot 2^{-(s-1)}$ passes, which is near 0.75 extra passes for $P=3$ branching at $s=3$. That compares with up to 64 passes for 75% overlap. The branch depth $s$ and count $P$ are the knobs we sweep.
The third component is phase disagreement as uncertainty. From the $P$ branches we compute a per-voxel variance map. We test whether that map ranks erroneous voxels on shifted scanners. We also test whether it can stand in for the multi-pass augmentation "susceptibility" term that Ouyang et al. feed into their spatially varying temperature. This speaks directly to the tension in the file: single-pass methods delete the variance signal that current uncertainty estimators rely on.
The fourth component is training-regime crossing. Networks are trained either on standard truncated patches or with the halo regime, where patches carry real surrounding context rather than padding. This lets us separate "full context helps" from "full context mismatches training." The 3D-UNet is the backbone because Ji et al. found it beat Transformer alternatives on AMOS: 89.04% versus 86.32% mean Dice on CT.
We deliberately do not pursue caching inside test-time adaptation ensembles such as InTEnt or OptTTA, although the file flags this gap. InTEnt's candidates differ in batch-normalization statistics from the first normalization layer onward (Dong et al.), and OptTTA's candidates differ at the input (Tomar et al.). Almost nothing downstream is shareable in either case, so the reuse opportunity there is small. Sliding-window overlap, by contrast, shares nearly everything.
Baselines
The naive reference is non-overlapping single-pass tiling with the same trained network. Hadlich et al. report Dice of 81.02% for this on AutoPET, and we re-run it on every dataset. It fixes the floor for both accuracy and compute.
The strongest existing competitor is 75% overlap with Gaussian center weighting, the configuration that reached 83.83% Dice on AutoPET in the same paper. It is the default in practice, and it is precisely the redundancy we aim to eliminate. We also sweep 25% and 50% overlap to trace the full accuracy–compute curve. Cube size, network weights, blending, and post-processing are identical across all inference variants, and all run on the same GPU with wall-clock time, FLOPs, and peak memory logged. AMOS reports 89.04% CT mean Dice for 3D-UNet (Ji et al.), but the file does not state the overlap setting, so we reproduce it rather than quote it as the overlap baseline. No reported result was found on Lacuna for the wall-clock cost of 75% overlap on these benchmarks. Beating this baseline means matching its Dice within the seed-to-seed standard deviation in domain, and exceeding it on unseen-vendor MRI, at no more than 2 network-pass equivalents of compute.
Whole-volume single-pass inference on a large-memory GPU is the exactness check for halo caching. The two must agree to numerical precision; if they do not, the implementation is wrong.
The ablations isolate the key ingredient. Halo caching alone tests context repair without any ensemble. Phase branching without halos isolates diversity. Branching the same number of deep passes at the zero offset with dropout active controls for "any stochastic ensemble" versus phase-specific diversity. Varying branch depth $s$ measures how much of the gain lives in deep layers.
For uncertainty, the baselines are local temperature scaling and the method of Ouyang et al.. That paper reports Expected Calibration Error of 14.48% (local temperature scaling) versus 12.78% (theirs) under simulated artifacts, and 16.78% versus 15.37% under the bSSFP→LGE sequence shift. Beating them means reaching comparable calibration error with our phase variance substituted for their augmentation susceptibility, at lower pass count.
Why not just…?
Why not just drop overlap to zero or 25%? The measured gap of 81.02% versus 83.83% (Hadlich et al.) says zero overlap loses real accuracy. The overlap sweep will show whether a modest overlap already captures most of the gain. If 25% overlap plus Gaussian weighting matches 75% under shift, then context repair is cheap and our ensemble arm is unnecessary. We would report that plainly.
Why not just run the whole volume in one pass on a bigger GPU? This is the real threat to the context-repair arm. Halo caching is literally equivalent to it, and its value is bounded memory, not accuracy. The more interesting question is whether whole-volume inference is even good for a patch-trained model, given the patch-to-volume collapse documented by Moturu and Joshi. The training-regime cross decides it. If whole-volume inference underperforms truncated overlap for patch-trained nets but not for halo-trained nets, the overlap gain was compensating for a train–test context mismatch.
Why not just train a shift-equivariant (anti-aliased) network so no phase ensemble is needed? This is also a genuine threat. If blur-before-downsample training removes phase sensitivity and matches overlap accuracy under shift, phase ensembling is moot. However, the uncertainty signal would then vanish too. We include an anti-aliased 3D-UNet trained with the identical budget. If it wins on Dice, the contribution narrows to a diagnosis plus halo caching, and we accept that outcome.
Why not just add flip-based test-time augmentation or test-time adaptation? Flips cost a full pass each, and OptTTA-style augmentation search costs several (Tomar et al.). Test-time adaptation from single images is unstable on small targets (Dong et al.; Janouskova et al.). We compare flip ensembling at matched compute. We treat InTEnt as complementary, stacking it on top of each inference scheme, and do not treat it as a rival.
Proposed Experiments
The primary testbed is AMOS (Ji et al.). We train on CT and a subset of MRI vendors, then evaluate in domain and on held-out MRI vendors, where drops exceed 20%. We report mean Dice and small-organ Dice (adrenals, duodenum, pancreas), because those are where boundary truncation should hurt most. AutoPET (Hadlich et al.) provides the published overlap numbers to reproduce, plus whole-body false-positive counts in regions of normal physiological uptake. For calibration we use a 2D variant on ACDC with simulated artifacts and on MS-CMRSeg bSSFP→LGE, matching Ouyang et al.. There we report Expected Calibration Error and the error-detection area under the ROC curve for the phase-variance map.
The first decisive check is a dose–response curve: overlap gain (75% minus zero overlap) plotted against distribution distance, going from in-domain CT to seen-vendor MRI to unseen-vendor MRI. Claim 2 predicts that halo caching closes most of the gap in domain but a shrinking fraction out of domain, with phase branching closing the remainder. The second check is the training-regime cross described above. The third is whether phase variance correlates with errors on unseen vendors at least as strongly as augmentation-based variance while using fewer passes.
The claims fail cleanly under two outcomes. If halo caching plus phase branching at 2 pass-equivalents falls short of 75% overlap on unseen-vendor MRI, claim 2 is false. If the zero-offset dropout control matches phase branching, phase is not the operative source of diversity.
References
- AutoPET Challenge 2023: Sliding Window-based Optimization of U-Net
- A fast learning algorithm for image segmentation with max-pooling convolutional networks
- Volume-based Performance not Guaranteed by Promising Patch-based Results in Medical Imaging
- Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Medical Image Segmentation with InTEnt: Integrated Entropy Weighting for Single Image Test-Time Adaptation
- OptTTA: Learnable Test-Time Augmentation for Source-Free Medical Image Segmentation Under Domain Shift
- Single Image Test-Time Adaptation for Segmentation