Many medical image models still work one 2D slice at a time, including adaptations of 2D foundation models such as the Segment Anything Model (SAM). When they segment a CT or MRI volume this way, they lose the depth context a radiologist gets by scrolling through slices. An organ is found on one slice and missed on the next. Boundaries jitter along the depth axis. Labels can even swap sides. We propose to restore that lost context after the fact. The tool is a generative prior over 3D label maps, not images: a model that has learned what plausible multi-organ anatomy looks like in 3D, and that can rewrite only the parts of a stacked slice-wise prediction it finds implausible.
The prior uses a discrete latent representation. A vector-quantized autoencoder (an encoder whose outputs are snapped to the nearest entry of a learned dictionary, or "codebook") turns a 3D label volume into a short grid of integer tokens. A masked Transformer then learns which token arrangements are anatomically likely. Because the prior sees only organ shapes, not intensities, it should transfer across scanners and modalities, which is where image-trained 3D networks break. Because it is a discrete sequence model, it gives an explicit likelihood that can flag and localize segmentation failures. The two-sentence version: train a token-level model of 3D anatomy on label maps alone, then use it to find and resample the regions where a slice-wise segmenter lost volumetric context. We expect this to matter most when the segmenter is applied to a modality or scanner it was not trained on.
Core Claim
Volumetric context loss in slice-wise segmentation is mostly a shape error, not an appearance error. Shape is the thing a label-space prior can learn without paired images and carry across modalities. Concretely, we claim three things. A discrete masked prior trained only on CT organ label maps can raise the 3D Dice and inter-slice consistency of slice-wise predictions on CT and on held-out MRI. It can do so without retraining the segmenter. And the prior's per-token likelihood identifies the corrupted regions well enough to predict which cases fail.
The closest existing work uses discrete 3D tokens in medicine for other purposes. Realistic morphology-preserving generative modelling of the brain pairs a 3D VQ-VAE with an autoregressive Transformer to generate brain MRIs. Its precursor Neuromorphologicaly-preserving Volumetric data encoding using VQ-VAE shows such tokenizers preserve morphology (MS-SSIM 0.991–0.998). Transformer-based out-of-distribution detection for clinically safe segmentation uses token likelihood over image volumes as an external gatekeeper that rejects scans before segmentation. All three model images, and none of them edits a segmentation.
The method we borrow comes from shape modeling. AutoSDF shows that a prior over patch-wise VQ tokens can complete a shape from an arbitrary observed subset of patches. We move that idea from ShapeNet objects to multi-organ label volumes and use it for repair rather than completion. Moving to label space is the key departure. It is also why our likelihood signal may succeed where Graham et al.'s image-space likelihood could not detect left–right flips (AUC 0.55): a swapped left/right kidney label is a gross error in label space even when the image looks symmetric.
Research Program
Tokenizer. The first component is a 3D VQ-VAE over one-hot multi-organ label volumes. The label maps are resampled to a fixed physical grid in millimetres, so that each token covers the same anatomical extent regardless of slice thickness. This is a modest answer to the anisotropy problem that CARVE emphasizes for clinical scans.
Quantization loses information. 3DILG reports that VQ degrades reconstruction relative to continuous latents, and thin organs such as the adrenal glands are most at risk. For this reason the tokenizer's encode-decode Dice on ground-truth labels is measured per organ before anything else. That number is an upper bound on what the prior can ever write back.
Prior. The second component is a bidirectional masked Transformer over the token grid, in the masked-modeling style of Point-BERT, trained by randomly masking tokens and predicting them. We use a bidirectional model rather than a raster-order autoregressive one. Repair requires conditioning on arbitrary surviving context, and the file notes that raster ordering imposes artificial causality on space. An optional variant also conditions each token on image features pooled from the frozen 2D encoder, so that resampled regions follow the actual scan rather than the textbook average. Comparing the two variants tests the "textbook bias" concern raised by Your other Left!.
Repair procedure. Inference proceeds in five steps. First, any slice-wise segmenter produces masks, which are stacked and encoded into tokens. Second, each token gets a suspicion score that combines the prior's conditional probability of that token, the segmenter's softmax entropy, and disagreement with adjacent slices. Third, the most suspicious fraction of tokens is masked and iteratively resampled. Fourth, the decoded label volume replaces the original prediction only inside masked regions, and the segmenter's own output is kept elsewhere so that quantization cannot damage regions that were already correct. Fifth, several samples yield a voxel-wise uncertainty map, and the sequence log-likelihood yields a per-case quality score.
The prior is trained only on AMOS CT label maps from AMOS. It is then applied unchanged to CT, to held-out MRI, and to BTCV. The prior never sees MRI.
Baselines
The do-nothing reference is the slice-wise segmenter's stacked output. We use two segmenters: a SAM-based 2D model in the style of ProMISe-type adaptation, and a 2D U-Net, both trained on AMOS CT. Every repair method starts from identical predictions, so any gain is attributable to the repair step. The prior beats this reference if it improves mean Dice and our inter-slice consistency metrics on both CT and MRI without lowering Dice on any organ by more than the tokenizer's measured ceiling gap.
The cheap post-processing baseline applies largest-connected-component filtering per organ followed by a median filter along depth. It removes speckle and jitter at near-zero cost, so it is the honest bar for "context repair." It is tuned on the same validation scans as the prior's masking fraction. Our method must exceed it on Dice and, more tellingly, on missing-segment and laterality errors that filtering cannot fix.
The strongest learned competitor for slice-wise models is SAM3D. It stacks frozen SAM ViT-B slice embeddings and decodes them with a 1.88M-parameter 3D CNN, and it reports 79.56% DSC on Synapse and 90.41% on ACDC. It learns volumetric context directly from paired image volumes. We train it on AMOS CT with the same data and a matched GPU-hour budget, then test on CT and MRI.
The strongest native-3D reference is the 3D-UNet that AMOS reports at 89.04% mDice on its CT test set, above Swin-UNETR's 86.32%. The same paper reports that performance drops by over 20% on unseen MRI scanners. We do not expect to beat a 3D-UNet in-domain on CT, and we will not claim to. We instead apply our prior on top of the 3D-UNet's MRI predictions, to test whether shape repair recovers part of that cross-scanner drop.
The key ablation replaces the VQ bottleneck with a continuous denoising autoencoder of the same size, trained on the same corrupted-label objective. It isolates whether discreteness buys anything beyond "a learned 3D shape prior." A second ablation removes image conditioning, and a third replaces suspicion-based masking with random masking of the same fraction.
Why not just…?
Why not just train a 3D network like nnU-Net and avoid the slice-wise problem entirely? In-domain, this is the right answer; AMOS shows a well-tuned 3D-UNet beating Transformers on CT. The proposal targets a different regime. Many deployed and foundation-model pipelines are slice-wise, and image-trained 3D networks themselves lose over 20% under unseen MRI scanners. A label-only prior is plausibly modality-agnostic. The experiment that settles this applies the prior to 3D-UNet MRI outputs. If it adds nothing there, the claim shrinks to "a fix for 2D models only."
Why not just add SAM3D's small 3D decoder? This is a real threat on CT, where SAM3D learns depth context from paired data at low cost. Its weakness is that the decoder is trained on CT images and inherits their appearance shift on MRI. Our prior needs no MRI images or labels. The CT-trained SAM3D versus CT-trained repair comparison on held-out AMOS MRI decides it. If SAM3D wins there too, the discrete prior is not worth its complexity for segmentation, and only its likelihood use remains.
Why not just smooth along depth and keep the largest component? Filtering fixes jitter but cannot restore a missing stretch of pancreas or correct a left/right kidney swap, because it has no notion of anatomy. We report error categories separately, covering fragmentation, missing segments, and laterality swaps, so that a filtering win on jitter alone cannot be mistaken for equivalence.
Why not just use a continuous shape prior, since quantization loses detail? 3DILG is direct evidence that continuous latents reconstruct better. If the continuous denoising-autoencoder ablation matches the discrete prior on Dice, we will say so. The discrete model must then justify itself through what the continuous one lacks: an exact token likelihood for flagging cases, as Graham et al. exploit, and native arbitrary-subset resampling. These are measured below, not assumed.
Proposed Experiments
The main experiment trains the tokenizer and prior on AMOS CT training labels and evaluates on three sets: AMOS CT test, AMOS MRI (never seen by the prior), and BTCV CT as an external dataset. Metrics are mean and per-organ Dice. Inter-slice consistency is measured as the fraction of organs whose prediction splits into multiple disconnected depth runs, plus boundary displacement between adjacent slices.
A controlled anisotropy test downsamples CT to 3 mm and 5 mm slice spacing before the 2D segmenter runs. We expect the repair gain to grow with slice thickness. If it does not, the "context loss" interpretation is wrong.
The likelihood experiment computes the correlation between per-case sequence log-likelihood and Dice. It also reports area under the ROC curve for flagging cases below a Dice threshold, and compares this against segmenter softmax entropy. A synthetic laterality test swaps left and right kidney and adrenal labels in otherwise correct predictions and measures detection AUC. This is the label-space counterpart of the image-flip failure (AUC 0.55) in Transformer-based out-of-distribution detection for clinically safe segmentation.
Three checks are decisive. First, the tokenizer ceiling must exceed the slice-wise segmenter's Dice on the small organs; otherwise repair can only hurt them. Second, the repaired CT-trained 2D model must beat CT-trained SAM3D on AMOS MRI. Third, the discrete prior must beat the continuous ablation on at least the likelihood-based failure detection, even if not on Dice. Failing the second check would mean volumetric context is better learned from images than from shape. Failing the third would mean discreteness is incidental, and we would report the work as a continuous shape-prior result.
References
- Realistic morphology-preserving generative modelling of the brain
- Neuromorphologicaly-preserving Volumetric data encoding using VQ-VAE
- Transformer-based out-of-distribution detection for clinically safe segmentation
- AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation
- CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
- 3DILG: Irregular Latent Grids for 3D Generative Modeling
- Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- ProMISe: Promptable Medical Image Segmentation using SAM
- SAM3D: Segment Anything Model in Volumetric Medical Images