Large image segmentation models trained on 2D photographs, such as the Segment Anything Model (SAM), are increasingly applied to CT and MRI scans one slice at a time. Treating each slice alone discards what happens between slices, and the result is fragmented masks: an organ is found on one slice, missed on the next, and found again after that. The usual fix is to insert small trainable "adapter" branches that let neighbouring slices exchange information. The trouble is that a freshly initialized branch adds random noise to features the frozen model depends on. Med-Tuning reports exactly this failure: its volumetric adapters degrade performance when placed in early encoder layers, and the authors had to choose insertion depths by hand.
We propose a cross-slice adapter whose final output projection starts at exactly zero, so that on the first training step the adapted model is the slice-wise 2D model. The branch takes as input the change in features between adjacent slices, and it is modulated by the physical slice spacing of the scan. Because the model starts as an exact copy of the 2D model, an adapter can be placed in every layer without risk, and how far each layer's projection grows away from zero shows where volumetric context is actually needed. The central scientific question is whether the early-layer degradation reported for volumetric adapters is an artifact of initialization, which this design removes, or a real incompatibility between early 2D features and mixing across slices, which it would expose.
Core Claim
We make three claims, each of which can be tested and could fail. The first is that once a cross-slice adapter is zero-initialized, inserting it at every encoder depth becomes safe: the insertion-depth sensitivity documented in Med-Tuning should disappear. If it persists, early features really do resist mixing across slices. The second is that the difference between adjacent slices' features is a better input to the branch than raw neighbour features. The reason is that the frozen 2D encoder already describes each slice well, so what is missing is how anatomy moves along the depth axis. This is the signal ConResNet showed to be informative at boundaries, although it extracted that signal with an extra decoder trained from scratch, at an 18.6% increase in compute. The third is that conditioning the branch on physical slice spacing lets a single adapter handle both thin-slice and thick-slice protocols, including spacings not seen in training. Uneven spacing across axes (anisotropy) is a known open problem that no paper in our source material evaluates for 2D-to-3D adaptation.
Mechanism
Let $h_\ell(z)$ be the frozen encoder's token features at layer $\ell$ for slice $z$, and let $\Delta z$ be the scan's slice spacing in millimetres. The adapter updates each slice's features as
$$h_\ell(z) \leftarrow h_\ell(z) + P_\ell\, \phi_\ell\big(h_\ell(z{+}1)-h_\ell(z),\; h_\ell(z)-h_\ell(z{-}1);\; \Delta z\big).$$
Here $\phi_\ell$ is a small network: a depth-wise convolution along the slice axis over the two difference maps. Its intermediate activations are scaled and shifted by a function of $\log \Delta z$. $P_\ell$ is a linear output projection initialized to zero. Only $P_\ell$ starts at zero. $\phi_\ell$ keeps a standard random initialization, so the gradient with respect to $P_\ell$ is nonzero from the first step. This avoids the "dead branch" failure that arises when every part of the branch is zeroed. The cost scales linearly with the number of slices and touches only two neighbours per slice, which sidesteps the cubic cost of full 3D attention.
The design takes its protective principle from zero-initialized fusion in other settings. ZECO uses zero-weight 1×1×1 convolutions to inject masks into a frozen 3D diffusion model. SPAD uses zero-initialized cross-view attention in a 2D diffusion model. G-CUT3R uses zero convolutions to add depth priors to a reconstruction network. LLaMA-Adapter uses zero-gated attention in a language model. All four apply the idea to generation, reconstruction or language. None addresses discriminative 2D-to-3D transfer, and none conditions the injected signal on acquisition geometry. ZECO in particular starts from an already-3D pretrained model, so it does not face the 2D-to-3D gap at all.
Research Program
The backbone is SAM's image encoder, kept frozen, with a trainable mask decoder. Adapters are added at every encoder block. We run three studies.
The first is an insertion-depth sweep that reproduces the Med-Tuning phenomenon. Adapters are placed only in the first quarter of the blocks, only in the last quarter, or in all blocks. Each placement is run under five initializations: standard random, small-variance random, random with learning-rate warmup, zero output projection, and zeroed output projection plus a zeroed scalar gate. The five-way comparison separates the effect of "the model starts as an exact copy" from "the branch simply starts small".
The second study tracks the norm of each $P_\ell$ over training, separately for each dataset and each spacing. The aim is to map where cross-slice information enters. We expect to see more growth in late layers and for thin slices. If the pattern turns out to be flat, that would argue against the premise that depth needs differ by layer.
The third study tests spacing robustness. We take near-isotropic CT and resample it to 1.5, 3 and 5 mm slice thickness. We train on a subset of these spacings and test on held-out ones. Three variants are compared: spacing conditioning, no conditioning, and conditioning with a shuffled spacing value.
Baselines
The naive reference point is frozen SAM with the same trainable decoder applied slice by slice, followed by 3D connected-component filtering and morphological closing. This is the "do nothing volumetric" option, and it shows how much of any gain simple post-processing could have recovered. Shi et al. report that SAM used without any training reaches a mean Dice of 0.2105 on AMOS CT, but our naive baseline adds a trained decoder, so we will measure it ourselves.
The strongest adapter competitor is Medical SAM Adapter. It adds a depth-transposed attention branch alongside the frozen SAM encoder and generates adapter weights through a hypernetwork, updating about 2% of parameters. On BTCV it reports a 34.8% Dice improvement over vanilla SAM and a 2.9% improvement over Swin-UNETR. The research file gives only these relative figures, not an absolute Dice. We will retrain it on our splits using its released design, matching its trainable-parameter budget of roughly 13M, and give both methods the same hyperparameter search budget and the same number of GPU-hours. Beating it means higher Dice, lower 95th-percentile Hausdorff distance, and fewer inter-slice presence flips on BTCV and KiTS, averaged over three seeds, with non-overlapping standard errors.
Med-Tuning is the second adapter baseline. It combines 3D depth-wise convolution with an FFT branch in an adapter. It reports up to +17.13% tumour Dice over full fine-tuning on KiTS 2019. We run it with its published insertion depths and also with adapters in every block. The every-block setting is where we expect it to fail and our method to hold.
The native-3D reference is SAM-Med3D, which is pretrained on 22,000 volumes. It reports a 60.12% Dice improvement over the original SAM. SAM-Med3D-MoE reports 48.9% Dice on SPPIN against 32.3% for the foundation model. As a from-scratch 3D reference we include 3D UX-Net, which reports 0.900 Dice on AMOS after transfer from FLARE, and 0.938 Dice on FLARE 2021. On Pancreas-CT, ConResNet reports 86.06% mean Dice.
The ablations that isolate the key ingredients are as follows. The same adapter with random initialization isolates the effect of the zero start. Raw neighbour features instead of differences isolate the difference input. Removing spacing conditioning isolates the anisotropy handling. A LoRA-style update that works within each slice, with no cross-slice path, isolates volumetric context from extra capacity.
Why not just…?
Why not just use random initialization with a learning-rate warmup? This is the most serious threat. Warmup limits how much the weights move early on, but it does not stop random branch outputs from shifting the frozen features on the very first forward pass. The Med-Tuning early-layer degradation is circumstantial evidence that this matters. Circumstantial evidence is not proof, so the insertion-depth sweep gives warmup and small-variance initialization the same tuning budget as the zero start. If warmup matches our method at every insertion depth, we will report that zero initialization is not the active ingredient and reframe the contribution around the difference input and spacing conditioning.
Why not just fine-tune with LoRA, which already zero-initializes one factor? LoRA's zero start protects the same property, but it adapts each slice independently and so adds no information across slices. The within-slice LoRA ablation measures exactly this. If LoRA matches our method, the gains came from capacity rather than volumetric context, and the slice-consistency metrics should reveal it.
Why not just use a native 3D foundation model? SAM-Med3D's reported 60.12% gain over SAM is large. Its limits are that it depends on 22,000 curated volumes and, according to SAM-Med3D-MoE, forgets general classes when specialized. We compare against it directly on its reported weak categories and on SPPIN. We treat matching it on rare structures with about 2% trainable parameters as success, and we do not claim to beat it overall.
Why not segment slice by slice and smooth in 3D afterwards? Roth et al. show that slice-wise predictions give fragmented 3D shapes. Smoothing can remove isolated specks but cannot restore an organ that was missed on an entire slice. The naive baseline measures this gap directly, particularly on thin structures such as the pancreatic tail.
Proposed Experiments
We evaluate on BTCV, KiTS 2019, AMOS 2022, NIH Pancreas-CT and BraTS, plus the SPPIN challenge data for low-data transfer. Metrics are Dice, 95th-percentile Hausdorff distance, and two continuity measures. The first continuity measure is the per-structure rate of slices where a structure is present in the ground truth but absent in the prediction between two slices where it is found. The second is the number of 3D connected components compared with the ground truth.
The decisive check is the insertion-depth sweep. If the random-initialization adapter degrades when placed early while the zero-initialized one does not, the first claim holds. A second decisive check is the held-out-spacing test. If spacing conditioning beats the unconditioned adapter at unseen slice thicknesses, and shuffling the spacing value removes that advantage, the third claim holds. The difference-input claim holds only if differences beat raw neighbour features on the continuity metrics, and not only on Dice.
References
- Med-Tuning: Exploring Parameter-Efficient Transfer Learning for Medical Volumetric Segmentation
- Inter-Slice Context Residual Learning for 3D Medical Image Segmentation
- ZECO: ZeroFusion Guided 3D MRI Conditional Generation
- SPAD: Spatially Aware Multi-View Diffusers
- G-CUT3R: Guided 3D Reconstruction with Camera and Depth Prior Integration
- LLaMA-Adapter: Efficient Fine-tuning of Large Language Models with Zero-initialized Attention
- Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation
- Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
- SAM-Med3D: Towards General-Purpose Segmentation Models for Volumetric Medical Images
- SAM-Med3D-MoE: Towards a Non-Forgetting Segment Anything Model via Mixture of Experts for 3D Medical Image Segmentation
- 3D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentation
- Deep convolutional networks for pancreas segmentation in CT imaging