Teaching Flat Slice Models Depth by Transferring a 3D Model's Corrections

Most strong image models see one 2D picture at a time. When they read a CT or MRI scan, they handle it as a stack of separate slices, so they never learn that an organ on slice 40 should usually still be there on slice 41, or that it has ended by slice 60. The resulting masks are jagged and broken across slices. Native 3D models avoid this, but they cost much more memory, need large voxel-annotated datasets, and usually resample anisotropic scans (scans with fine in-plane pixels but thick slices) onto isotropic grids, which blurs fine in-plane detail. This proposal borrows a recipe from cross-domain recommendation, where a small learned function translates a user's preferences from a data-rich platform to a data-poor one. We treat a 3D model's depth-aware judgment as the "rich domain" and a frozen 2D foundation segmenter as the "sparse domain." We then learn a small non-linear map that turns a slice's 2D features, together with its neighbors' features and the slice spacing, into a correction added inside the frozen 2D model.

In two sentences: a small network trained on a few paired "anchor" volumes learns to predict, from a 2D model's own slice features, the adjustment a 3D model would make. The 2D model then gains volumetric consistency at inference without being retrained and without running the 3D model. The bet is that this correction must be non-linear and must be conditioned on slice spacing, because anatomy appears, splits and vanishes across depth in non-monotonic ways. The bet would fail if a plain linear map, post-hoc smoothing, or an equally sized adapter trained directly on labels did just as well.

Core Claim

Three prior lines of work supply the method. In the embedding-and-mapping approach to cross-domain recommendation (Cross-Domain Recommendation: An Embedding and Mapping Approach), each domain's representation is learned separately, and a multi-layer perceptron is then fit on overlapping "anchor" users to translate one space into the other. Two related studies argue the translation must be non-linear. Cross-Domain Collaborative Filtering with Review Text shows U-shaped cross-domain relations that it handles with radial basis function kernels. Out-of-Town Recommendation with Travel Intention Modeling conditions the map on a context variable, the travel intention, that changes how preferences drift. Activation Space Interventions Can Be Transferred Between Large Language Models carries the same idea into neural networks. It trains a ReLU autoencoder to map hidden activations from a 1B to a 3B language model and injects the mapped steering vectors into the larger model's residual stream. That non-linear map clearly beat affine maps, and it fully removed a code-vulnerability backdoor.

None of these papers deals with spatial depth, and the activation-transfer work is limited to text token streams. We claim that out-of-plane context in medical volumes can be transferred the same way: as an additive intervention learned from anchor data and injected into a frozen 2D model. Concretely, let $h_z$ be the frozen 2D encoder's feature map for slice $z$, $\Delta$ the physical slice spacing in millimeters, and $g_\phi$ a small non-linear network with parameters $\phi$. The corrected feature is

$$\tilde h_z = h_z + g_\phi\big(h_{z-k},\dots,h_{z+k};\ \Delta\big),$$

where $k$ is the number of neighboring slices on each side. We train $g_\phi$ on anchor volumes so that the frozen 2D decoder, fed $\tilde h_z$, reproduces the 3D teacher's mask on that slice; ground-truth labels enter as a secondary term when they exist. The 2D model keeps its native in-plane resolution. Only $g_\phi$ sees depth, and only $\Delta$ tells it how far apart the slices are.

The non-linearity hypothesis is specific and testable. The correct correction at an organ's end slice depends on whether the cross-section is shrinking or growing across neighbors and on how far away those neighbors are. Strong evidence on both neighbors means "keep." Strong evidence on one side and none on the other means "probably ending," with a strength that depends on $\Delta$. That is a sign-dependent, non-monotonic function of neighbor features, the same kind of structure the review-text paper found in user preferences. A linear map cannot represent it. This is our account of the over-propagation that Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2 documents when video segmenters run along the depth axis.

Research Program

The 2D model is SAM with a medical decoder. We inject the correction in one of two places: into the image embedding, or as extra decoder tokens in the manner of the Incremental Pattern Shifting module from ProMISe: Promptable Medical Image Segmentation using SAM. ProMISe uses a roughly 1.3M-parameter network to push a frozen SAM toward medical shapes on single 2D images. We reuse its injection mechanism, but our tokens carry information from neighboring slices and slice spacing rather than a per-image shape bias.

The teacher is SAM-Med3D: Towards General-Purpose Segmentation Models for Volumetric Medical Images. It is a native 3D promptable segmenter trained on SA-Med3D-140K, and it reports a 60.12% Dice improvement over 2D SAM with one point prompt. We use it only during training, on thin-slice anchor volumes where its isotropic assumption causes the least harm.

The mapper $g_\phi$ is a small depthwise-attention network over $2k+1$ slice features, with $\Delta$ fed through a learned embedding. We compare three versions under a matched parameter budget: a non-linear MLP, an RBF-kernel version, and a purely linear version.

The study has three analyses beyond headline accuracy. First, we vary the number of anchor volumes, which plays the role of the overlap-set size that limits embedding-and-mapping methods. This shows where the non-linear mapper begins to overfit, a failure the recommendation literature flags. Second, we break down errors by slice position: interior slices, the first and last slices containing the structure, and bifurcations. The non-linearity claim should show up mostly at structure ends. Third, we probe whether the learned correction resembles a steering direction. Concretely, we test whether a correction learned for one organ class transfers to unseen classes, which is the medical analogue of how far an activation intervention generalizes.

Baselines

The naive reference point is the frozen 2D SAM-family model run slice by slice with identical prompts and no correction. It defines the volumetric context loss we aim to recover.

The classic free fix is post-hoc 3D Gaussian smoothing of the stacked probability maps, as in Deep convolutional networks for pancreas segmentation in CT imaging. There it raised mean Dice from 56% to 68% on 82 pancreas CT volumes. We tune the kernel width separately per slice spacing on the same validation split. Beating it means higher Dice and lower boundary error at structure ends, where smoothing is expected to blur or extend masks.

The strongest slice-propagation method is SLM-SAM 2 from Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2. It adds short- and long-term memory banks and a ConvNeXt fuser to SAM 2 and reports an average Dice gain of 0.14 over SAM 2 on AMOS and other datasets. The file gives no absolute AMOS Dice for it, so we re-run it with its 1-volume and 5-volume annotation setups. We give our method the same number of annotated volumes as anchors.

The strongest native 3D comparisons are SAM-Med3D itself, run directly at test time, and a supervised 3D-UNet. AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation reports 89.04% mDice for 3D-UNet on the AMOS CT test set, against 86.32% for Swin-UNetr. We do not expect a label-scarce frozen-2D method to beat a fully supervised 3D-UNet on in-distribution thin-slice CT. On that setting, the honest success criterion is closing most of the gap between the uncorrected 2D model and 3D-UNet while training under 1% of its parameters. Lacuna has no reported SAM-Med3D result on AMOS, so we measure it ourselves. Beating SAM-Med3D means exceeding it on thick-slice and unseen-scanner test sets at lower memory.

Two ablations isolate the key ingredients. The first is a linear (affine) mapper with the same inputs and the same injection point. If it matches the non-linear mapper, the transfer framing still stands but the non-linearity claim fails. The second is the identical $g_\phi$, trained only on ground-truth labels with no 3D teacher, which is effectively a 2.5D adapter. It isolates whether transferring the teacher's judgment adds anything beyond adding parameters that see depth. We also test the mapper without $\Delta$ to isolate spacing conditioning. Every method gets the same anchor volumes, the same number of trainable parameters where applicable, and the same hyperparameter search budget, which we set at a fixed number of trials.

Why not just…?

Why not just smooth the 2D outputs in 3D? Smoothing is a low-pass filter applied after the decision. It cannot recover a structure missed on several consecutive slices, and it spreads masks past an organ's true end. The pancreas study plateaued at 68% ± 10% Dice even with smoothing (Deep convolutional networks for pancreas segmentation in CT imaging). Our per-position error breakdown tests directly whether the learned correction wins mainly at structure ends. If smoothing matches us everywhere, the proposal adds nothing.

Why not just use a linear map between slices or spaces? The evidence favors non-linear maps for both preferences (Cross-Domain Collaborative Filtering with Review Text) and model activations (Activation Space Interventions Can Be Transferred Between Large Language Models). Neither result concerns spatial depth, though, so this alternative is a real threat. The matched linear ablation settles it. We commit in advance to reporting the non-linearity claim as false if the linear mapper comes within noise across seeds at structure-end slices.

Why not just train the same small adapter on labels and skip the 3D teacher? This is the most serious threat. A 2.5D adapter with access to neighboring slices may learn depth consistency on its own when labels are plentiful. The no-teacher ablation answers it. We expect the teacher to matter most when anchor volumes are few and on unseen scanners, where AMOS reports drops of over 20% for MRI (AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation). If the label-only adapter matches us even with one to five volumes, we will narrow the claim to "a cheap 2.5D adapter suffices" and drop the transfer framing.

Why not just run SAM-Med3D or SLM-SAM 2 directly? Both are included as baselines. Our expected advantage is specific: thick, anisotropic scans that SAM-Med3D must resample and that break the smooth-motion assumption of video propagation, plus lower memory use than a 3D vision transformer. If either model matches us on thick-slice and cross-scanner tests, the remaining advantage is efficiency alone, and we will report it that way.

Proposed Experiments

The main benchmark is AMOS, with 500 CT and 100 MRI scans, 15 organs, and eight scanner models. We train mappers on thin-slice CT anchors in budgets of 1, 5 and 20 volumes. We test on held-out CT, on held-out CT made artificially thick by dropping slices to 3 mm and 5 mm spacing, and on MRI from scanners never seen in training. The pancreas CT cohort from Roth et al. serves as the hard, highly variable single-organ case, if it can be accessed; its reported 56% and 68% Dice give a fixed reference. AbdomenMRI provides a second MRI check, where Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining reports 0.7760 Dice for a fully trained 2D network with ImageNet pretraining. We report it as context rather than as a matched competitor.

Metrics are Dice and 95th-percentile Hausdorff distance, computed both overall and on structure-end slices. We also count over-propagated slices, where a structure is predicted beyond its true extent, and record peak GPU memory.

The experiment that decides the project is the ablation pair on thick-slice and cross-scanner test sets. We test whether the non-linear, spacing-conditioned mapper beats the linear mapper and the label-only adapter by margins larger than seed variance, and whether those gains concentrate at structure ends. If they do, depth context behaves like a transferable non-linear intervention, and the same recipe can be tried on medical vision-language models, which Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images shows score near chance on slice-level spatial questions. If they do not, we will have shown cleanly that simpler depth adapters suffice, which is still a useful negative result.