Clinical scans such as CT and MRI are stacks of 2D slices. The decisions that matter clinically are usually fine-grained ("stage T2 or T3?", "mild or moderate?") rather than "sick or healthy," and they often hinge on a handful of slices where a tumor breaches a boundary. Current systems either look at slices one at a time and lose the context along the stack, or compress the whole stack into a single vector that dilutes those few decisive slices.
We propose a different kind of classifier. We learn a probability model of what each slice's features should look like given its neighbors and a candidate severity level. We then diagnose a volume by asking which severity level makes the whole stack most probable. Because the model scores every slice under every severity, it also reports, without any lesion detector, which slices argue for a higher grade. The bet is that this generative route is more robust than direct discriminative training in the regime these tasks live in: a few hundred labeled patients, ordered grades, and large drops when moving to a new hospital.
Core Claim
A model that estimates densities over continuous slice features, conditioned on a continuous severity variable, can be read as a Bayes classifier over ordered disease stages. Continuous here means real-valued features rather than discrete codebook tokens. We claim this classifier will match the best discriminative volumetric models on in-distribution staging accuracy. We expect it to beat them in three places. First, severity staging on held-out sites. Second, low-label regimes, since the unconditional part of the density can be trained on unlabeled scans. Third, localizing the slices that carry diagnostic evidence.
The mechanism combines three things from the file. The first is the continuous-token density head of GIVT, which predicts a Gaussian mixture over real-valued tokens instead of a softmax over a codebook. The second is the within-frame/across-frame attention split of VideoMAR, with its causal time mask replaced by bidirectional masking, because the depth axis has no arrow of time. The third is the ordered, anchored label geometry that CMCNet and DeepTag show matters for fine-grained diagnosis.
What Exists and What Is Different
Three lines of work motivate the gap.
The first line handles volumetric context with discriminative networks. Moving from 2D to 3D stages rectal cancer (T2 vs. T3) with a late-fusion 3D CNN, bilinear pooling and a triplet loss, and reports AUC 0.831. MFM-Geom inflates a 2D vision transformer to prostate MRI and pools tokens into a covariance matrix for a binary cancer label. Both output a label and nothing about which slices mattered. Both are binary, and neither uses unlabeled volumes.
The second line recovers volumetric context by cascading tools. 3DMedAgent routes an agent through segmentation, heatmaps and single-slice inspection, storing text notes. The unified 2D DeepLesion framework detects, crops, then segments. Both pipelines fail completely when the first stage misses. Transcribing images into text also invites the language-driven errors documented by Wolf et al.: vision-language models were at chance on relative position in CT until organ names were replaced with abstract markers.
The third line supplies the continuous-density machinery. GIVT and VideoMAR use it only for natural-image and video generation, never as a classifier and never on medical stacks.
The closest single idea, "VideoMAR for CT," would be a causal generator of slices. We do not generate slices, and we do not use a causal mask. We use the conditional density purely as a scoring function over an ordered label.
Research Program
Each slice $z$ of a volume is encoded by a frozen 2D medical encoder (BiomedCLIP, the backbone used by CMCNet and MFM-Geom). A fixed linear projection, PCA fit on training slices, reduces its patch tokens to a small set of low-dimensional continuous vectors $x_z$. The dimension is kept in the tens, which is the regime where GIVT's mixture head is stable.
A transformer attends bidirectionally within a slice and across slices. Each slice gets a position embedding in physical millimetres, so varying slice thickness is represented explicitly rather than interpolated away. During training, contiguous blocks of slices are masked. A Gaussian-mixture head predicts the masked tokens given the visible slices and a severity code $s$. The code $s$ is a scalar on $[0,1]$ with Fourier-feature embedding, and ordered stages are placed at fixed points on it. With some probability, $s$ is replaced by a null token, as in classifier-free guidance. This lets the same network learn an unconditional density, and unlabeled volumes can train that part.
At test time we score a volume with the masked pseudo-likelihood
$$ L(s) = \sum_{z=1}^{D} \log p_\theta\!\left(x_z \mid x_{\setminus z},\, s\right), $$
where $D$ is the number of slices, $x_{\setminus z}$ is the stack with slice $z$ masked, and $p_\theta$ is the predicted mixture density. The severity posterior is $p(s \mid x) \propto p(s)\,\exp(L(s))$, evaluated on a grid of $s$. The predicted stage is its mode, and its spread is an abstention signal in the spirit of DeepTag's abstain layer.
The per-slice term $\ell_z(s_{\text{high}}) - \ell_z(s_{\text{low}})$ gives a depth-wise evidence profile. Here $\ell_z(s)$ is the summand in $L(s)$ for slice $z$. The profile shows which slices favour the higher grade.
We will study three questions:
- Does the continuous severity code actually learn ordered structure? Held-out intermediate grades should receive intermediate posteriors.
- How many masked-slice passes are needed? The cost is roughly $D$ forward passes per candidate grid point, and we will measure how much block masking reduces that.
- Does the density's multimodality matter? We compare against a single-Gaussian head.
Baselines
The naive reference is a per-slice classifier on the same frozen BiomedCLIP features, with slice probabilities averaged across the volume. This is the setup whose limits Roth et al. exposed for segmentation (Dice 56% raw, 68% after 3D smoothing). We also include a variant with Gaussian smoothing along depth before pooling. Beating it is the minimum bar and would say little on its own.
On rectal cancer, the strongest existing method is the late-fusion 3D CNN with bilinear pooling and triplet loss from Moving from 2D to 3D. It reports AUC 0.831, accuracy 79.3% and T2 recall 62.1% on its 567-patient cohort, against radiologist AUCs of 0.64 to 0.73. It is the strongest competitor because it is purpose-built for the same anisotropic MRI staging task. We will retrain it on our exact splits with the same augmentation and the same hyperparameter-search budget in GPU-hours. Beating it means matching its AUC within confidence intervals and exceeding its T2 recall, the minority class it handles worst.
On prostate MRI, the reference is MFM-Geom. It reports AUC-PR 90.67 with 10% of PI-CAI training data (against 82.37 for a class-token baseline), and 90.61 AUC-PR on external PROSTATE158 for its unimodal variant. We compare at the same 10% label fraction. The prediction is that our advantage grows as the label fraction shrinks, because unlabeled PI-CAI volumes feed the null-conditioned density.
The decisive ablation is a discriminative twin. It uses the same frozen tokens, the same bidirectional transformer, the same millimetre position embeddings and the same compute. Its mixture head is replaced by an ordinal classification head trained with cross-entropy plus CMCNet's center-margin loss toward text anchors of each stage. This isolates the one thing we claim is new, which is classifying through a conditional density rather than a decision boundary.
A second ablation keeps generative pretraining but swaps the mixture head for mean-squared-error regression of masked tokens, then fine-tunes discriminatively. This separates "the density is the classifier" from "masked pretraining helps."
Why not just…?
Why not just train a discriminative slice transformer on the same features with an ordinal loss? This is the real threat. On in-distribution accuracy, we expect it to tie. Our claim therefore does not rest on in-distribution accuracy. It rests on three things. The first is held-out-site severity staging, where the neonatal echocardiography study saw F1 fall from 0.84 to 0.63 with a discriminative 3D CNN. The second is performance at 5–10% labels. The third is the evidence profile. If the discriminative twin ties on external PROSTATE158 and in low-label runs, the honest conclusion is that the density buys interpretability only, and we will report it that way.
Why not just use a 3D CNN such as nnU-Net or a 3D ResNet? The file offers two pieces of evidence against it. Late fusion beat full 3D convolution on anisotropic rectal MRI (Moving from 2D to 3D), and native 3D foundation models scored 37–39% on DeepTumorVQA versus 66% for 3DMedAgent. Still, we include the strongest 3D CNN in the file as a baseline rather than dismissing the family.
Why not just pretrain with masked feature regression and fine-tune? Regression with mean-squared error is a single-Gaussian density with fixed variance. It cannot represent the fact that a slice adjacent to a tumor margin has several plausible appearances. The second ablation tests exactly this. If regression pretraining plus a discriminative head matches us, the mixture head is unnecessary.
Why not just run an agent over slices, as 3DMedAgent does? The agent stores its evidence as text and depends on an early localization step. Wolf et al. show that language priors override pixel evidence on CT. Our evidence profile stays in feature space and needs no detector. We test whether it finds the right slices; we do not assume it.
Proposed Experiments
The primary staging experiment uses the rectal-cancer MRI cohort from Moving from 2D to 3D, subject to data access. It spans 7 scanners, and we hold out whole scanner groups as pseudo-external sites. We report AUC, balanced accuracy, per-stage recall and expected calibration error.
The second experiment trains on PI-CAI and tests on PROSTATE158, at label fractions of 5%, 10% and 100%. We use the binary clinically-significant-cancer label for comparability with MFM-Geom. Where case-level grade information in the release permits, we add an ordered grade task. If it does not, we report no grading result.
For evidence localization, we apply the unconditional and lesion-conditioned model to DeepLesion. We measure whether the peak of the evidence profile falls on annotated lesion slices, and compare against its detector's slice selection (reported detection mAP50 70.1%). No reported result was found in the file for slice-level evidence localization by any volumetric classifier, so this comparison is new ground.
Three checks decide the question. First, on held-out sites and at 10% labels, the density classifier must beat the discriminative twin on balanced accuracy or T2 recall, with non-overlapping bootstrap intervals. If it cannot, the core claim fails. Second, posteriors for held-out intermediate grades must be ordered: training on the extreme stages alone should still place the middle stage between them. Third, abstaining on the 10% of cases with the widest posteriors must raise accuracy more than abstaining by softmax confidence from the twin.
References
- GIVT: Generative Infinite-Vocabulary Transformers
- VideoMAR: Autoregressive Video Generation with Continuous Tokens
- CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification
- DeepTag: inferring diagnoses from veterinary clinical notes
- Moving from 2D to 3D: Volumetric Medical Image Classification for Rectal Cancer Staging
- A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification
- 3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis
- A Unified 2D Framework for DeepLesion Detection, Segmentation and Short Report Generation
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
- Deep convolutional networks for pancreas segmentation in CT imaging
- Deep Learning Based Prediction of Pulmonary Hypertension in Newborns Using Echocardiograms