Networks that segment CT and MRI scans lose information along the depth axis. Most of them process 2D slices one at a time or cut the volume into small 3D blocks. They do this because the full volume does not fit in memory, and as a result they cannot follow an organ or lesion from one slice to the next. Researchers have treated this loss of through-slice ("volumetric") context as an architecture question: which slices should the network look at? It is better understood as a memory-allocation question. Suppose the scan is fed through an accelerator one slice at a time, with each layer keeping a small on-chip buffer of the most recent slices. Each layer can then see exactly as many neighbouring slices as its buffer holds. Every slice of depth context has an exact, countable cost in on-chip memory (SRAM), and that cost differs from layer to layer, because early layers have large feature maps and deep layers have small ones.
We propose to make this trade explicit. Each layer gets an integer depth extent: how many slices its convolution spans, and with what spacing. These integers are searched under a hard SRAM budget, with the memory cost computed exactly rather than relaxed into a smooth surrogate. The output is a curve of segmentation quality against on-chip memory, for both overlap accuracy and slice-to-slice consistency. Each point on the curve comes with a concrete allocation saying where in the network depth context is worth paying for. The search variables are integers and the memory cost is a step function, so this is a non-differentiable hardware problem. Its decision variable, though, is the amount of volumetric context.
Core Claim
Under a fixed on-chip memory budget for slice-streaming inference, the way through-slice context is distributed across layers matters more than the network's operator family. A searched, per-layer integer allocation of depth extent will beat uniform 2D, uniform 3D, pseudo-3D, and V-NAS-style choices at equal memory, in Dice and especially in slice-to-slice consistency. The gap should be largest on thick-slice (anisotropic) scans and on small targets.
Two things are new here. The first is the formulation: depth context becomes a discrete resource with an exact hardware price, so memory never has to be approximated by a softmax-weighted expectation. The second is the evidence: a memory–quality Pareto curve for 3D medical segmentation, together with an analysis of which layers the optimum spends depth on.
Relation to Prior Work
V-NAS is the closest work and supplies the problem setting. It chooses among 2D, 3D, and pseudo-3D convolutions at each layer with a DARTS-style softmax relaxation and reports 85.15% mean Dice on NIH Pancreas. It optimizes accuracy only and has no memory or hardware cost. Its choice is also coarse: depth extent is 1 or 3, never "5 slices at spacing 2 in the bottleneck." We keep its insight that anisotropic volumes want asymmetric per-layer depth handling, but we price that depth exactly and search over it under a budget.
Three papers constrain how the search should be done.
- ZARTS shows that DARTS's gradient approximations bias the architecture objective and can shift its minimum toward degenerate choices. This argues against inheriting V-NAS's relaxation and in favour of derivative-free search over discrete candidates.
- Auto-NBA jointly searches networks, bit-widths, and accelerator buffer and tiling parameters on 2D classification, sampling hard discrete hardware paths. We reuse its idea of evaluating hardware cost on sampled discrete configurations. We replace its 2D pipeline template with a slice-streaming one in which buffer depth along the scan axis is the object being searched.
- DISeR uses reinforcement learning over a hardware grammar but retrains a perception network inside every reward evaluation. We avoid that cost by training one weight-sharing network once and searching over it.
EffConv supplies a training mechanism. It learns kernel sizes by softly masking kernel borders. We use a hard, sampled truncation along the depth axis only, so that one network supports every depth extent.
CARVE motivates the problem. In 3D multimodal models, it finds that adjacent slices are highly redundant while diagnostic cues are sparse and concentrated in one or two slices. That observation suggests uniform depth context is wasteful. However, CARVE is a training-free token selector for language-model pipelines and is out of scope here.
Research Program
The accelerator model streams slices along the scan axis. In-plane, each slice is cut into a fixed number of tiles with halos. Layer $l$ keeps a first-in-first-out buffer of recent feature slices whose length is set by its depth kernel. If layer $l$ uses depth extent $d_l$ and depth dilation $r_l$, it needs $(d_l-1)r_l+1$ buffered slices. The on-chip memory of an allocation $a=\{(d_l,r_l)\}$ is
$$M(a)=\sum_{l} \big((d_l-1)r_l+1\big)\, T_l\, C_l\, b,$$
where $T_l$ is the in-plane tile area at layer $l$, $C_l$ its channel count, and $b$ the bits per activation. The search problem is to maximize validation Dice subject to $M(a)\le B$ for a memory budget $B$.
Because the whole volume streams through, there are no block boundaries along the depth axis. The only form of context loss left is the limited depth reach of each layer. That reach is exactly what the allocation controls, and its total is the network's through-slice receptive field. The cost model also counts off-chip traffic for any buffer that spills and gives a cycle-level latency estimate. We calibrate it against a measured FPGA implementation of a few allocations, in the style of Auto-NBA's analytical models.
Training uses one U-Net-shaped network whose depth kernels are allocated at maximum extent. At each step a random allocation is sampled, and depth taps outside it are zeroed, so the shared weights learn to work under any truncation. Sampling is biased toward allocations near the budgets of interest. Search then runs on the frozen network. The main searcher is a zero-order evolutionary search that computes $M(a)$ exactly and ranks candidates by validation Dice. As a cheaper alternative we also run a greedy method that adds depth one step at a time at the layer with the best measured Dice gain per byte. The top candidates from either searcher are retrained from scratch before reporting.
Dice alone hides the failure we care about, so we also measure depth-consistency error. This is the fraction of adjacent slice pairs in which the predicted mask creates or deletes a connected component that the ground truth does not. It directly counts the "organ present on slice $z$, gone on slice $z+1$" failure that the pancreas CT work describes.
Baselines
Naive reference. A 2D per-slice U-Net and a uniform 3D U-Net, each configured nnU-Net style as in the Medical Segmentation Decathlon. Each is mapped onto the same streaming accelerator at each budget. Where the 3D model does not fit, it falls back to the largest uniform depth extent that does. Uniform pseudo-3D, with depth extent 3 everywhere, is a third fixed point.
Strongest existing method. V-NAS is the strongest competitor because it is the only method in the file that searches per-layer depth handling for 3D medical segmentation. It reports 85.15% mean Dice on NIH Pancreas at about 1.2 V100 GPU-days of search per fold, plus 55.27% on MSD lung tumours and 37.78% on MSD pancreas tumours. Those numbers carry no memory constraint. We therefore run it two ways. First, as published, which gives an unconstrained reference. Second, with a differentiable penalty on expected memory under its softmax weights, with the penalty weight swept to cover each budget; this is the fair constrained version. Both runs use the same data splits, the same augmentation, the same GPU-day budget as our training plus search, and the same retraining of the final architecture. Beating V-NAS means higher Dice and lower depth-consistency error than constrained V-NAS at equal exact SRAM across budgets, on at least NIH Pancreas and two MSD tasks. Matching unconstrained V-NAS at a fraction of its memory also counts.
Ablations. The first isolates the key ingredient by comparing our allocations against random allocations at the same budget and against fixed rule-based ones: all depth in the bottleneck, all depth in the first layers, or depth proportional to feature-map cheapness. The second compares the exact-cost evolutionary search against the greedy searcher on the same shared network. The third removes dilation from the search space.
Why not just…?
Why not just run nnU-Net and shrink the patch until it fits? nnU-Net's configuration rules choose between 2D and 3D and size patches to fit GPU memory. They do not decide where depth context is spent, and they say nothing about on-chip budgets. The decathlon results show hard volumetric targets remain weak; the decathlon paper reports 0.21 Dice for pancreas tumours. Our naive-reference rows are exactly this approach. If they lie on our Pareto curve, the claim that allocation matters is false, and we will report that.
Why not just add a memory penalty to V-NAS? This is the main threat, and it is our constrained baseline. Two things argue against it. ZARTS shows that DARTS-style relaxation biases the architecture objective. And an expected memory under softmax weights is not the memory of the configuration that is finally discretized, so a solution can satisfy the relaxed budget and still violate the real one. Our experiments report the exact SRAM of V-NAS's discretized architectures, which settles whether this mismatch is large in practice.
Why not just use a simple greedy rule instead of search? This might plausibly work as well. The problem is close to a knapsack in which each layer's depth step has a cost and a Dice gain. For that reason greedy is built into the program as a searcher on the same shared network. If greedy matches evolutionary search, we adopt it as the cheaper method, and the contribution becomes the streaming formulation, the exact cost, and the allocation findings. Evolutionary search only earns its place if interactions between layers make greedy measurably worse at tight budgets.
Why not just use a 3D transformer or Mamba for global context? UNETR reports gains such as +1.238% average Dice on BTCV, but it notes that finer patch sizes run into GPU memory limits. Swin-UMamba scans in 2D directions and has no explicit depth path. Neither is designed for a fixed on-chip budget. We include UNETR on BTCV only as an accuracy reference with its memory footprint reported, not as a streaming competitor.
Proposed Experiments
Datasets are NIH Pancreas with 4-fold cross-validation, matching V-NAS, together with MSD lung tumour, pancreas, spleen, and liver, and BTCV. For each, we trace Dice and depth-consistency error against an SRAM budget swept over roughly an order of magnitude, alongside calibrated latency and off-chip traffic.
Three checks are decisive. First, at matched exact SRAM, searched allocations must beat both uniform allocations and constrained V-NAS. Without that, the core claim fails. Second, an anisotropy test: we resample the same scans to thicker slices, about 5 mm, by dropping slices, and re-search. If the method behaves as the formulation predicts, the optimal allocations should shift toward larger depth spacing and depth placed in deep layers. Gains over uniform allocation should grow with slice thickness. Third, transfer: an allocation searched on NIH Pancreas is evaluated on MSD lung tumours, following V-NAS's transfer protocol, to test whether allocations reflect general structure or overfit one organ. For the top three allocations, we validate the cost model on an FPGA implementation and report how far measured latency and memory deviate from the model's predictions.
References
- V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation
- ZARTS: On Zero-order Optimization for Neural Architecture Search
- Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators
- DISeR: Designing Imaging Systems with Reinforcement Learning
- EffConv: Efficient Learning of Kernel Sizes for Convolution Layers of CNNs
- CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
- Deep convolutional networks for pancreas segmentation in CT imaging
- The Medical Segmentation Decathlon
- UNETR: Transformers for 3D Medical Image Segmentation
- Swin-UMamba: Mamba-Based UNet with ImageNet-Based Pretraining