Large image segmentation models such as the Segment Anything Model (SAM) are trained on flat photographs. When they are run on a CT or MRI scan one slice at a time, each slice is segmented with no knowledge of the slices above and below it. The resulting 3D masks flicker: an organ can be outlined on one slice and vanish on the next, and boundaries look jagged in side views. The usual fixes either blur the stacked output, which erases small tumors, or bolt on full 3D layers, which are expensive. We propose a third option. A small trainable module produces a per-slice "steering" vector that adjusts how the frozen 2D model behaves on that slice. During training we add a penalty that keeps neighboring slices' steering vectors close, and we relax that penalty wherever the frozen model's own features change sharply between slices. Smoothness is therefore enforced on how the model is told to behave, not on what it outputs, and it switches off where the anatomy itself is changing.
The bet is simple. Consistency across slices and preservation of small lesions conflict when smoothness is imposed on images or predictions. That conflict should mostly disappear when smoothness is imposed on a low-dimensional control signal whose neighbor graph comes from the data. If the bet is right, a parameter-light 2D adapter should match or beat post-hoc 3D aggregation on volumetric overlap. It should also clearly beat output-level smoothing on lesions that span only one to three slices.
Core Claim
Let $m_z \in \mathbb{R}^d$ be the steering vector for slice $z$. It is produced by a small network $g_\phi$ from pooled features of slice $z$ and its two neighbors, and it modulates the frozen encoder and decoder through low-rank feature scaling. Let $F(z)$ be the frozen encoder's pooled feature for slice $z$, with gradients stopped. The training objective is
$$\mathcal{L} = \mathcal{L}_{\text{seg}} + \lambda \sum_{z} a_z \,\|m_z - m_{z+1}\|^2, \qquad a_z = \exp\!\big(-\|F(z) - F(z+1)\| / \tau\big).$$
Here $\mathcal{L}_{\text{seg}}$ is the usual per-slice Dice plus cross-entropy loss, $\lambda$ sets the penalty strength, and $\tau$ is a temperature. The penalty is a graph-Laplacian smoothness term along a chain of slices. Its edge weights $a_z$ come from the inter-slice feature change.
This term combines two pieces of prior work. The Laplacian form follows Q-MKL, which smoothed classifier weights over a fixed 3D voxel grid of brain scans. The edge signal is the inter-slice feature residual that ConResNet used as an attention map. Q-MKL leaves open how to build the graph, and in our design the frozen backbone supplies it. ConResNet uses residuals as activations inside a full 3D network; we use them only to decide where smoothness applies.
The approach does not regularize slice-indexed weights. Absolute slice index means different anatomy in different patients, so a weight tied to slice 40 would encode nothing consistent. It also avoids 3D convolutions inside the backbone.
Relation to Prior Work
Several papers motivate the gap. Roth et al. showed that independent 2D predictions do not assemble into coherent organs, and that 3D Gaussian smoothing of the outputs raised pancreas Dice from 56% to 68%. PanSAM runs SAM per slice and then trains a separate 3D convolutional aggregator over the stacked probabilities, reaching 87.01% Dice on NIH Pancreas-CT. Both papers repair consistency after the 2D model has already made its decisions. We instead shape the 2D model's per-slice behavior during training.
Other papers constrain the design. Inter-Slice Consistent Stochasticity reports that Total Variation across slices over-smooths and erases small tumors. That paper keeps its own smoothing training-free, interpolating diffusion noise between two fixed end slices, and the authors concede this cannot represent structures that appear mid-volume. Our gate is the direct response to the over-smoothing finding, and our steering vectors are learned rather than interpolated. Learnable Pooling Regions adds a further reason to expect benefit: spatial parameters optimized without smoothness scatter into noise, while adjacency penalties produce coherent shapes.
Two papers supply the adaptation recipe. ProMISe adapts frozen SAM with about 1.3M trainable decoder parameters but stays purely 2D. 3DSAM-adapter gets depth context by inserting 3D depth-wise convolution adapters and a 3D decoder, tuning 16.96% of parameters on fixed sub-volumes. Our adapter is far smaller and processes slices, so memory should stay close to 2D inference.
Research Program
We will build $g_\phi$ as a 1D convolution over pooled slice descriptors. It feeds low-rank scale-and-shift modulation into a frozen SAM ViT-B encoder and its mask decoder, adding well under 1% of parameters. Training samples contiguous slabs of slices so the penalty sees real neighbors. At inference the whole stack is processed slice by slice, with a single pass over pooled descriptors.
The analysis has three parts. We will plot $a_z$ against ground-truth boundary and lesion locations to check that the gate opens where anatomy changes. We will measure how far $m_z$ jumps between neighboring slices with and without the penalty. We will also bin lesions by how many slices they span, so that any loss on small lesions is visible rather than averaged away.
Baselines
The naive reference is frozen SAM with the same prompts, run slice by slice. We report it with no post-processing and with 3D Gaussian smoothing of the probabilities as in Roth et al.. Their 68% Dice comes from an older ConvNet, so we rerun the Gaussian step on our SAM outputs rather than quote it.
The strongest prompt-free competitor on pancreas is PanSAM, which reports 87.01% Dice and 81.42% Jaccard on NIH Pancreas-CT, and 83.20% zero-shot Dice on AbdomenCT-1K. We will use PanSAM's automatic prompt generator for all pancreas methods, so that prompt quality is held fixed; PanSAM attributes a 2-point Dice loss to that generator. We will reproduce PanSAM on our splits and compare against both its reported and reproduced numbers. Beating it means at least matching its volumetric Dice with fewer trainable parameters and one training stage, and also showing a lower surface distance or less slice-to-slice flicker.
ConResNet is the strongest native-3D reference. It reports 86.06% Dice on NIH Pancreas-CT and 83.15% enhancing-tumor Dice on BraTS 2018, at an 18.6% GFLOPs increase. For promptable tumor segmentation, 3DSAM-adapter reports only relative gains over nnU-Net, for example +29.87% Dice on MSD-Pancreas tumors. No absolute reported result was found on Lacuna for 3DSAM-adapter on MSD-Pancreas, so we will rerun it with the same single-click prompts and report memory alongside Dice.
All trained methods get the same data splits, the same prompt source, the same number of training steps, and the same hyperparameter search budget of roughly 20 trials each.
Three ablations isolate the key ingredient. The first removes the penalty ($\lambda = 0$). The second keeps the penalty but makes it uniform ($a_z = 1$), which is a plain Laplacian. The third, and most important, keeps the gated weights $a_z$ but moves the penalty from $m_z$ to the predicted masks, so it acts on outputs instead.
Why not just…?
Why not just smooth the output probabilities with a 3D Gaussian or a small 3D aggregator? Gaussian smoothing helps but blurs low-contrast boundaries, and it left Roth et al. at 68% Dice. A trained aggregator like PanSAM's is much stronger, and it is the main threat on whole-organ Dice. Our claim against it is narrower: it adds a second training stage and cannot recover a lesion that the 2D stage missed. The size-binned lesion recall comparison will settle whether that limitation matters in practice.
Why not just add an inter-slice consistency loss on the predicted masks? This is the most serious alternative, because it is simpler than our method and also acts during training. Total Variation across slices is known to erase small tumors ISCS, and an output penalty shares that risk even with gating. The third ablation tests exactly this, with the same gated weights. If output-level gated smoothness matches us on lesions spanning one to three slices, then the control-space argument is wrong, and we would report the simpler loss as the contribution.
Why not just use 3D adapters as in 3DSAM-adapter? These adapters work well. However, they train on fixed 3D sub-volumes with higher memory, and they impose no smoothness on what they learn. We will report Dice per gigabyte of peak memory. If 3DSAM-adapter wins clearly on Dice at comparable memory, our method is mainly a cheap alternative rather than a better one, and we will say so.
Proposed Experiments
The main experiments run on NIH Pancreas-CT, AbdomenCT-1K zero-shot transfer, BraTS 2018, and MSD-Pancreas tumors. Metrics are Dice, Jaccard, average symmetric surface distance, a flicker score (one minus the Dice between consecutive slice masks within the true extent of the structure), lesion recall binned by slice extent, trainable parameters, and peak memory.
The decisive check compares our method against the output-level gated penalty and against PanSAM on the lesion bins spanning one to three slices. Gated control-space smoothness should reduce flicker relative to the unregularized adapter without lowering small-lesion recall. The uniform-penalty ablation should show exactly that drop in small-lesion recall, reproducing the over-smoothing effect at the level of the control signal.
A second check tests the gate itself. We swap the residual-derived $a_z$ for random weights with the same distribution. If performance holds, the data-derived graph adds nothing.
References
- Q-MKL: Matrix-induced Regularization in Multi-Kernel Learning with Applications to Neuroimaging
- Inter-Slice Context Residual Learning for 3D Medical Image Segmentation
- Deep convolutional networks for pancreas segmentation in CT imaging
- PanSAM: Zero-Shot, Prompt-Free Pancreas Segmentation in CT Imaging
- Improving 2D Diffusion Models for 3D Medical Imaging with Inter‑Slice Consistent Stochasticity
- Learnable Pooling Regions for Image Classification
- ProMISe: Promptable Medical Image Segmentation using SAM
- 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation