Clinical CT and MRI scans are three-dimensional, but they are acquired and stored as stacks of 2D slices. The in-plane resolution is fine, while the gap between slices is coarse and varies from scanner to scanner (roughly 0.5 mm within a slice against 2.5–5 mm between slices). Most models either process slices one at a time, and so lose the out-of-plane context that tells a vessel apart from a nodule, or resample every scan onto a fixed grid and assume it is uniform. We propose a generative model that treats a scan as an unordered set of observations, each located at its true position in millimetres. The model is trained so that, given any subset of slices, it can generate the anatomy at any other physical location, one location at a time and in any order. The same model then handles thick-slice-to-thin-slice completion, filling in missing or corrupted slices, and reconstruction from a sparse mix of axial and coronal slices. None of these tasks needs its own retraining.
The ingredient that makes this possible comes from order-agnostic autoregressive models, meaning generators that factorize a joint distribution under a random ordering of positions rather than a fixed left-to-right or top-to-bottom one. RandAR showed that a standard decoder-only transformer can generate images in random orders if each image token is preceded by a "position instruction token" that names the 2D location to fill next. It matched raster-order generators on ImageNet and gained inpainting and outpainting for free. We move that idea from grid indices to continuous physical coordinates that carry the slice thickness. This one change is what we expect to make order-agnostic generation work under the anisotropy and protocol variability of real scans.
Core Claim
An order-agnostic generator conditioned on physical position, rather than on grid index, can complete CT volumes from arbitrary slice subsets. We expect it to generalize to slice spacings never seen in training, and to produce completions that are both anatomically coherent across depth and honestly uncertain where the observed slices do not determine the answer. Deterministic interpolators and one-shot masked reconstructors cannot provide both at once. Concretely, the model factorizes a scan $X$, viewed as tokens $x_1,\dots,x_n$ with physical positions $p_1,\dots,p_n$ in millimetres, as
$$P(X) = \prod_{i=1}^{n} P\big(x_{\sigma(i)} \mid p_{\sigma(i)},\, \{(x_{\sigma(j)}, p_{\sigma(j)})\}_{j<i}\big),$$
where $\sigma$ is a random permutation drawn during training. The target position $p_{\sigma(i)}$ is announced by a position instruction token before its content is generated. Because positions are real-valued millimetres rather than slice numbers, "the slice 7.5 mm below this one" means the same thing whether the scan was acquired at 1 mm or 5 mm spacing.
Relation to Prior Work
Two lines of work supply the method. The first is order-agnostic factorization. Neural Autoregressive Distribution Estimation trained one network over random orderings so that it implicitly handles any set of missing inputs. Probabilistically Masked Language Model showed that masked prediction with a uniformly random masking ratio optimizes the same permutation objective. Auto-Regressive Masked Diffusion Models introduced schedules that gradually increase the randomness of orderings during training. We reuse the masking-equivalence to train efficiently and the ramped-randomness curriculum to stabilize training. The Peng result also constrains us, and we return to that below.
The second line is spatial order-agnostic generation. RandAR works on 2D natural-image grids with 2D rotary position embeddings over integer indices. AutoSDF trains a random-order transformer over a small, uniform, cube-shaped grid of CAD shape tokens. It also warns that a tokenizer with a large receptive field leaks information from unobserved regions into observed tokens, which is why it uses a patch-local tokenizer. We keep AutoSDF's patch-local discipline but make each patch local to one slice. RandAR is built on integer grid positions and AutoSDF on uniform synthetic voxels; neither faces the problem we target, which is spacing that is uneven between axes and different from scan to scan.
On the volumetric side, several papers motivate the gap. Roth et al. documented jagged, fragmented 3D organ boundaries from independent 2D slice predictions. HounsWorld packs three adjacent slices into the colour channels of a pseudo-image, which gives only local 2.5D context, and it generates in a fixed causal order. CARVE shows that inter-slice redundancy is high but that small pathologies can live on one or two slices, so uniform subsampling is dangerous. CARVE, however, only prunes tokens and cannot synthesize anything. MedGMAE learns volumetric representations by masked reconstruction into 3D Gaussians. It is bidirectional and one-shot, and it has no sampling procedure. Our proposal differs from all of these because it defines a sample-able joint distribution over arbitrary physical locations of a real scan.
Research Program
Everything starts with the tokenizer. We train a vector-quantized autoencoder, which maps small image patches to entries of a learned codebook, on 2D in-plane patches of individual slices. Its receptive field is restricted to the patch, following AutoSDF's leakage argument, and it takes raw Hounsfield-unit intensities clipped to a wide window, so tissue density keeps its physical meaning. Because the tokenizer never looks across slices, it is indifferent to slice thickness. All cross-slice reasoning is left to the transformer.
Each token carries a physical position (x, y, z) in millimetres plus the slice thickness that produced it. We encode these with a 3D rotary embedding whose frequencies are defined per millimetre, so that relative offsets between tokens mean the same physical distance across scans. The thickness enters as an extra embedding on the position instruction token, so that a target token knows whether it represents a 1 mm or a 5 mm slab. Coronal and sagittal observations are simply tokens whose positions lie on different planes. Mixing acquisition planes therefore needs no architectural change.
Training uses random subsets and random orders within sub-volumes. With in-plane downsampling to 256×256 and 16-pixel patches, a single slice yields 256 tokens, so a window of about 32 slices fits in roughly 8,000 tokens. The data are the abdominal CT collections available in the file: AbdomenAtlas1.0Mini, as used by MedGMAE, plus AMOS and FLARE'22 training splits. Thin-slice scans are the source of ground truth. Thick-slice inputs are simulated by averaging over physical slabs of a chosen thickness rather than by dropping slices, so that the simulated inputs have the partial-volume blur real thick slices have. We deliberately hold out one spacing, 3 mm, from training.
At inference the model generates missing locations in a chosen order. We compare coarse-to-fine orders, which fill every fourth missing slice first, with random orders, and we vary how many tokens are decoded in parallel per step, following RandAR's schedule. Repeated sampling gives a set of plausible completions. The spread among them is the model's uncertainty map, and we evaluate it directly rather than treating it as a byproduct.
Baselines
The naive reference is linear or cubic interpolation along depth between observed slices. Every completion method must beat it, and it shows how much of the task is trivially recoverable.
The strongest discriminative competitor is a 3D U-Net trained to regress thin slices from thick inputs with an L1 loss. It uses the nnU-Net self-configuring recipe of nnU-Net, which dominated the Medical Segmentation Decathlon with a plain U-Net and careful automation. For this completion task there is no reported result found on Lacuna, so we will train it ourselves on the same scans, the same simulated inputs and the same GPU-hour budget, with an equal hyperparameter search. Beating it means matching or exceeding it on downstream segmentation Dice of completed volumes and clearly exceeding it on the calibration of uncertainty. We do not expect a generative model to win on PSNR, which rewards blurry means, and we will not claim that it does.
The strongest representation-learning competitor is MedGMAE. It reports gains over VoCo of 2.98% on AMOS and 5.06% on FLARE'22 Dice with 1% labels; the file gives only these relative gains, and no absolute Dice is reported on Lacuna. We compare against it in two ways. First, as a one-shot masked reconstructor of missing slices. Second, as a pretrained encoder fine-tuned for 1%-label segmentation on AMOS and FLARE'22 under identical fine-tuning budgets, where beating it means exceeding its Dice at the same label fraction.
The fixed-order generative competitor is HounsWorld-style pseudo-frame packing with causal slice-by-slice generation, retrained at matched size on our data.
Two ablations isolate the key ingredient. The first uses the identical architecture and data, but resamples every scan to isotropic spacing and uses integer grid-index rotary embeddings, as in RandAR extended to 3D. This isolates physical coordinates. The second uses the same trained network decoded in a single parallel step. This isolates the value of sequential sampling.
Why not just…?
Why not just train a 3D U-Net to regress the missing slices? This is the real threat for the average-case metrics, and it may win on PSNR. Its weakness is structural. A regressor trained with L1 loss outputs the conditional median, so when a small lesion could plausibly lie between two observed slices it either blurs it away or commits silently to one answer, and it gives no signal of which it did. CARVE's observation that pathologies can occupy one or two slices is exactly the regime where this matters. The lesion-in-gap experiment below settles the question. If the regressor's residuals predict its own errors as well as our sample spread predicts ours, the generative framing is not justified.
Why not just use a masked autoencoder, given that masking is equivalent to permutation training? Peng et al.'s equivalence holds for the training objective. At inference, predicting all missing tokens at once assumes they are independent given the observed ones. When a whole 5 mm gap is missing, that independence is false: neighbouring generated slices must agree with each other, not just with the observed slices. MedGMAE-style one-shot reconstruction should therefore show the z-axis incoherence Roth et al. described. The single-step-decoding ablation of our own network measures this directly. If one-step decoding matches multi-step decoding on boundary continuity and downstream Dice, then sequential generation is unnecessary, and we would report the result as a masked model with physical coordinates.
Why not just resample everything to isotropic voxels and use RandAR or AutoSDF in 3D? Resampling bakes an interpolation into the inputs before the model sees them, and it fixes a single grid. The model then never learns what a thick slab means physically. The held-out-spacing test is designed to expose this. If the grid-index ablation matches the physical-coordinate model at the unseen 3 mm spacing, our central claim is wrong.
Why not just pack adjacent slices into pseudo-frames as HounsWorld does? Pseudo-frames assume three contiguous, evenly spaced slices, and they generate causally. They have no way to express "observed at z = 0 and z = 5 mm, predict z = 2.5 mm", nor to accept a coronal slice. We expect them to be competitive for dense, regular stacks and to degrade sharply when inputs are sparse or come from mixed planes, which the any-subset experiment tests.
Decisive Experiments
The first experiment tests spacing generalization. We train on simulated 1, 2 and 5 mm thick inputs and test at the held-out 3 mm spacing on AMOS and FLARE'22 validation scans. Completed volumes are then segmented by a fixed nnU-Net trained on true thin-slice data, and we report the Dice gap relative to segmenting the ground-truth volume. The physical-coordinate model should show a clearly smaller gap than the grid-index ablation at 3 mm while being comparable at the training spacings.
The second experiment is the lesion-in-gap test, and it decides whether generation is worth the cost. We use scans with small annotated structures; the file does not name a lesion dataset, so we would use small organs such as the adrenal glands and gallbladder in AMOS as a stand-in. We withhold the slices where the structure is thinnest and measure two things. The first is whether the fraction of samples containing the structure tracks how inferable it is from the remaining context. The second is whether per-voxel coverage of sample intervals is calibrated, compared against the U-Net's residual-based uncertainty.
The third experiment tests any-subset reconstruction: given five random axial slices and two coronal slices, we reconstruct the abdomen. The metrics are organ Dice on the completed volume and a z-continuity score, defined as the fraction of slice-to-slice organ-mask transitions that are anatomically implausible. We compare against pseudo-frame causal generation, one-shot masked reconstruction, and interpolation.
As a secondary check on representation quality, we fine-tune the transformer's encoder on 1%-label AMOS and FLARE'22 segmentation against MedGMAE. The proposal stands or falls on the first two experiments. If physical coordinates do not beat grid indices at the unseen spacing, or if sample spread is no better calibrated than regressor residuals, we would stop.
References
- RandAR: Decoder-only Autoregressive Visual Generation in Random Orders
- Neural Autoregressive Distribution Estimation
- Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
- Auto-Regressive Masked Diffusion Models
- AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation
- Deep convolutional networks for pancreas segmentation in CT imaging
- HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulation
- CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
- MedGMAE: Gaussian Masked Autoencoders for Medical Volumetric Representation Learning
- nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation