Large 3D medical scans, such as whole-body CT, are too big to segment at full resolution in a single pass. A common fix is a two-stage pipeline. A cheap, low-resolution "coarse" pass looks at the whole scan and decides where the organs are. A "fine" network then processes only the selected high-resolution patches. The weak point is the handoff between the two stages. The coarse stage's beliefs about where boundaries lie come from features that have been pooled and resized many times, and those beliefs drift by several voxels. Current pipelines handle this drift in one of three ways. They paste the coarse map onto the fine patch as an extra input channel. They blend the coarse and fine predictions with fixed weights. Or they discard the coarse features altogether. None of them corrects the drift.
We propose to correct it with a learned displacement field, a per-voxel 3D vector saying where to look up the coarse features. This idea comes from feature-alignment work on 2D image pyramids. Our version differs in one important way: the field is defined in the scan's physical coordinate frame and samples from the entire retained global feature volume, not from a cropped copy. A fine patch can therefore pull semantic context from outside its own crop. That fixes boundary drift, and it also softens the known failure where a crop clips the organ. The central empirical question is whether this learned correction beats two cheaper fixes: exact coordinate bookkeeping, and letting the fine network absorb the drift itself.
Core Claim
In coarse-to-fine volumetric segmentation, the stage interface carries a measurable boundary misalignment. Its size grows with the coarse stage's downsampling factor. It hurts most for thin or small structures such as vertebrae, the pancreas and small organs. We claim that a content-aware warp of retained global decoder features into each patch's frame, allowed to sample beyond the crop, reduces boundary error near patch margins more than three alternatives. The first is exact physical-coordinate resampling. The second is auto-context concatenation. The third is the same warp restricted to the cropped region. We also make a falsifiable prediction: the benefit should shrink toward zero for large, blob-like organs and at mild downsampling factors. If it does not shrink, the gain is generic extra capacity, not alignment.
Relation to Prior Work
The alignment side supplies the mechanism and the motivating evidence. Semantic Flow for Fast and Accurate Scene Parsing predicts a 2D flow field between adjacent pyramid levels and warps coarse semantics toward true edges. FaPN does the same with deformable offsets at each 2× transition and reports gains over bilinear FPN, largest on small objects. Rethinking Features-Fused-Pyramid-Neck gives the sharpest motivating result: fusion without alignment can be worse than no fusion at all. FADE adds a constraint. Its decoder-driven gating becomes erratic under sparse supervision, and sparsely sampled patches arguably create exactly such a regime. All of these align adjacent levels inside one network over one continuous image. None aligns across a crop boundary between two separately run networks at very different resolution ratios.
The coarse-to-fine side supplies the setting and its defects. No More Sliding Window (NMSW) selects the top-K high-resolution patches with a differentiable sampler and merges local and global predictions with learnable per-class weights and bilinear resampling. It is the strongest efficient pipeline in the file. Its aggregation still treats the two stages as spatially rigid. Small Organ Segmentation in Whole-body MRI concatenates coarse probabilities into the fine input (auto-context), which assumes the two are already aligned. SegVol discards its zoom-out features entirely. ResDSN C2F shows that hard crops cannot recover clipped anatomy, and that perturbing the coarse stage can make the fine stage miss the organ completely.
The nearest hybrid is EdgeFlow. It uses a flow module between a coarse and a fine network, but it aligns user-click heatmaps with image features in 2D. It has no autonomous patch selection and no cross-resolution volumetric interface. C2FTrans moves attention windows onto uncertain boundaries, but entirely inside one model's token grid, with no physical cropping. Our proposal differs from all of these in where the alignment lives: at the global-to-patch interface, in physical coordinates, with out-of-crop sampling.
Research Program
We first build a diagnostic, because the claim is only worth pursuing if the misalignment is real. Using an NMSW-style pipeline trained on WORD and TotalSegmentator, we measure, for each structure, the signed surface displacement between the coarse stage's upsampled boundary and the ground-truth boundary. We repeat this at coarse downsampling factors of 2, 4 and 8. Separately, we measure how much of that displacement survives when the coarse map is resampled with exact physical-coordinate bookkeeping, that is, with consistent voxel-center conventions and correct crop origins. This separates bookkeeping error, which is trivially fixable, from learned semantic drift, which is what alignment targets. If the remaining drift is under about one fine-resolution voxel, the method is unnecessary and we would report that as the result.
The method keeps the global decoder's feature volume $G$ in memory after the coarse pass. That volume is small, since it is low resolution. For a fine patch, let $x$ denote a voxel's physical coordinate. A light 3D convolutional head takes the fine encoder features $F(x)$ together with $G$ resampled at $x$, and predicts a displacement $\Delta(x)$. The aligned global feature is
$$\tilde{G}(x) = G\big(x + \Delta(x)\big),$$
where $G(\cdot)$ is trilinear lookup in the full global volume. Because the lookup is in the global volume, $x+\Delta(x)$ may land outside the patch. $\tilde{G}$ is then fused with $F$ in the fine decoder, replacing auto-context concatenation and replacing NMSW's scalar-weighted blending. $\Delta$ is bounded by a tanh scaled to a few coarse voxels.
Top-K patches can overlap, and in a shared physical frame they should agree. We therefore add a consistency term over overlap regions $\Omega$:
$$\mathcal{L}_{\text{cons}} = \frac{1}{|\Omega|}\sum_{x\in\Omega}\lVert \Delta_i(x)-\Delta_j(x)\rVert^2,$$
where $\Delta_i$ and $\Delta_j$ are the displacements predicted from two overlapping patches $i$ and $j$. This term directly probes the tension, noted in the file, between discrete patch sampling and continuous flow fields. Given FADE's warning about sparse supervision, we also test an optional weak target for $\Delta$: the displacement that maps the coarse boundary onto the ground-truth boundary, computed from distance transforms. This tells us whether unsupervised warping is stable on its own.
The experiments use WORD and the TotalSegmentator organ and vertebrae subsets, where NMSW provides a matched pipeline. NIH Pancreas tests a small, low-contrast organ in the hard-crop setting of ResDSN. Besides Dice, we report three further measures. The first is normalized surface Dice and 95th-percentile surface distance. The second is a "seam error": surface error restricted to voxels within a few voxels of any patch margin. The third is a clipped-anatomy recall: the fraction of ground-truth foreground lying outside the coarse ROI that the final output still recovers. Compute is reported as multiply-accumulate operations and wall-clock latency.
Baselines
The naive reference is sliding-window inference at native resolution with the same fine backbone. It is the accuracy gold standard that NMSW compares against. NMSW reports its cost as 88.0 TMACs and 99.0 s per scan on an H100, against 8.00 TMACs and 8.3 s for NMSW (No More Sliding Window). Beating sliding window is not our goal. Matching its Dice at NMSW-level compute is.
The strongest existing method is NMSW itself, with global-local aggregation through learnable class weights. It is the most efficient published pipeline here and already outperforms sliding window on the TotalSegmentator organ task. The file gives qualitative Dice claims but no per-dataset Dice values: no reported result found on Lacuna for NMSW Dice on WORD or TotalSegmentator. We will therefore retrain it ourselves with the released configuration where possible. Retraining also guarantees identical backbones, patch count K, data splits and an equal tuning budget of the same number of hyperparameter trials for each method. Beating it means a statistically significant reduction in seam error and surface distance on thin structures, especially vertebrae, at no more than 10% added MACs, with no loss in overall Dice.
For hard-crop cascades on NIH Pancreas, the reference figure is ResDSN C2F's 84.59% DSC (Volumetric Medical Image Segmentation). We will compare against our reimplementation, not against that number alone.
Three further baselines isolate the key ingredient, and each gets the same parameter budget as our method. The first is auto-context concatenation as in Valindria et al., with exact physical-coordinate resampling. The second is "global features, no warp": $G$ resampled at $x$ and fused identically, which isolates the value of retaining features from the value of warping them. The third is "in-crop warp", the same displacement head but with sampling confined to the cropped region of $G$, as a direct 3D transplant of FaPN or SFNet would do. The gap between in-crop and cross-crop warping is the proposal's specific contribution.
Why not just…?
Why not just resample the coarse map correctly? Much of the apparent misalignment in cascades may be off-by-half-voxel bookkeeping rather than learned drift. That is exactly why the diagnostic phase measures drift after exact resampling, and why exact resampling is a baseline in every comparison. If exact resampling closes most of the seam error, the warp is not justified, and we would say so.
Why not just concatenate the coarse probabilities and let the fine network learn the shift? A 3D CNN with a receptive field of tens of voxels can in principle absorb a small, smooth shift. This is a real threat. Two pieces of evidence argue against relying on it. Rethinking Features-Fused-Pyramid-Neck found that unaligned fusion can be worse than none, and EdgeFlow found that raw mask priors trap refinement networks in coarse-stage errors. Concatenation also cannot reach outside the crop. The deciding test is the parameter-matched auto-context baseline across downsampling factors. Our claim predicts that the gap widens at factor 8 and on vertebrae. If it does not, concatenation suffices.
Why not just enlarge the crop margin or sample more patches? A wider margin or a larger K directly addresses clipping and seams, and costs no new modules. We will plot accuracy against MACs for margin and K sweeps, and ask whether the warp achieves a better accuracy at equal compute. If added margin reaches the same seam error at similar cost, the cross-crop argument fails.
Why not just use a bigger fine network? FaPN reports that ResNet-50 with alignment beat ResNet-101 with plain FPN, which suggests capacity is not the bottleneck. That evidence comes from 2D images, so we check it directly with a fine backbone widened to match our added parameters.
Decisive Checks
Three results would settle the question. The first is the drift diagnostic. Residual misalignment after exact resampling must exceed about one fine voxel for thin structures at factor 4 or above. If it does not, the project stops there as a negative result. The second is the cross-crop versus in-crop ablation on the TotalSegmentator vertebrae subset and on NIH Pancreas, measured by seam error and clipped-anatomy recall. This is where the out-of-crop sampling should matter most. The third is a check on the prediction that the warp's gain vanishes on large organs such as the liver. If gains instead appear uniformly across all structures, they are better explained by added capacity than by alignment.
As a secondary probe of the coarse-stage fragility documented by ResDSN C2F, we will translate the coarse ROIs by controlled amounts and test whether the bounded cross-crop warp recovers the shifted anatomy or amplifies the error.
References
- Semantic Flow for Fast and Accurate Scene Parsing
- FaPN: Feature-aligned Pyramid Network for Dense Image Prediction
- Rethinking Features-Fused-Pyramid-Neck for Object Detection
- FADE: A Task-Agnostic Upsampling Operator for Encoder-Decoder Architectures
- No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top-K Patch Sampling
- Small Organ Segmentation in Whole-Body MRI Using a Two-Stage FCN and Weighting Schemes
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation
- Volumetric Medical Image Segmentation: A 3D Deep Coarse-to-fine Framework and Its Adversarial Examples
- EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flow
- BATFormer: Towards Boundary-Aware Lightweight Transformer for Efficient Medical Image Segmentation