Patient-Relative Slice Residuals: Turning 3D Scans into Table Rows for In-Context Diagnosis

Tabular foundation models such as TabPFN can classify a new example in a single forward pass. They read a small labeled table as context and predict the query row with no gradient training. That makes them attractive for medical imaging cohorts with a few hundred labeled patients, where fine-tuning a 3D network overfits. The obstacle is that a CT or MRI scan is not a row. It is a stack of dozens to hundreds of slices, and the diagnostic signal often sits on only one or two of them. The usual way to make an image fit a table is to embed it with a frozen vision model and average. Averaging over a volume is exactly the operation that erases a small lesion. This proposal asks how to compress a variable-length stack of slice embeddings into a fixed-width row that a tabular in-context learner can use without losing the "needle."

The proposed answer is a summary of each scan relative to the patient's own anatomy. For every embedding dimension, we measure how far the most unusual slices deviate from that patient's typical slice, and we keep a few such outlier statistics as columns. We then select among those columns with mutual information, because tabular in-context learners are unusually hurt by uninformative columns. Clinical covariates are appended as additional columns. The bet is that the main bottleneck for in-context diagnosis on volumes is not the learner but how the volume is turned into a row. A row built to preserve isolated-slice deviations should hold up under lesion sparsity and slice dilution, conditions under which mean-pooled rows degrade.

Core Claim

Two results from the two research directions point in opposite directions. When TabPFN must drop rows to fit its context, uniform random sampling of training examples does as well as K-Means or CoreSet sketching (Scaling TabPFN). When a 3D model must drop slices to fit its budget, uniform reduction is harmful, because pathology is anisotropic and may live on one or two slices (CARVE). These results describe different axes: examples across patients versus slices within a patient. Within-patient compression therefore needs its own design, and none of the cited work has built that design for a tabular learner.

Our claim is specific. Take a frozen 2D encoder applied slice by slice and build each patient's row from patient-relative residual statistics rather than means. This preserves small-extent pathology. The resulting advantage over mean-pooled rows should grow as lesions occupy fewer slices and as scans contain more irrelevant slices. Combined with TabPFN in context, this should match or beat a fine-tuned slice transformer when the cohort has a few hundred labeled patients or fewer.

We expect the learner to matter less than the row. The design therefore tests the same rows under CatBoost and logistic regression. If those learners do as well as TabPFN, we will report the contribution as a representation result, not an in-context-learning result.

Research Program

Each scan is split into $S$ slices. Each slice passes through a frozen DINOv2 encoder, the same 2D encoder family used by the Medical Slice Transformer, giving embeddings $h_1,\dots,h_S \in \mathbb{R}^d$. Let $m$ be the per-dimension median over that patient's slices. We define slice residuals $r_s = h_s - m$ and compute, per dimension $j$,

$$ f_j = \Big(\ \bar h_j,\ \ \tfrac{1}{k}\textstyle\sum_{s \in \mathrm{top}_k(|r_{\cdot j}|)} r_{sj},\ \ \max_s |r_{sj}| \Big), $$

where $\bar h_j$ is the ordinary slice mean, $\mathrm{top}_k$ picks the $k$ slices with the largest absolute residual in dimension $j$, and $k$ is small (1 to 3). This is the slice-axis analogue of CARVE's cross-slice residual retrieval. CARVE uses residuals to decide which visual tokens a language model keeps. We use them to define the columns a tabular learner sees. Because the median is taken per patient, the residual measures how unusual a slice is relative to that patient's own anatomy rather than relative to the cohort. This should make the statistic less sensitive to body size, scanner, and field of view.

These statistics give $3d$ candidate columns, far beyond the 500-feature budget used in Images as Tables. We reduce them with mutual-information selection computed only on the context set. We prefer selection to PCA because Scaling TabPFN found TabPFN sensitive to noise features and found selection more critical for it than for boosted trees. We also suspect PCA fit on pooled features ranks rare residual directions low, since they carry little cohort-wide variance. As a second route past the feature ceiling, we test the lightweight learned input encoder and bagging from TabPFN Unleashed. Clinical covariates, where the dataset provides them, are appended as raw columns. The program builds the row-construction module, runs it on four volumetric cohorts across training-set sizes, and analyzes when and why it helps.

Baselines

The naive reference point is the direct volumetric extension of Images as Tables. That work extracts frozen DINOv3 vectors, compresses them with Incremental PCA to 500 dimensions, and runs TabPFN, outperforming a specialized detector in low-data regimes on 2D images. Our version uses mean-pooled slice embeddings, PCA to the same width, and TabPFN. It shares our encoder, learner, column budget, and context set, so it isolates exactly the row construction. We beat it if residual rows improve AUC on held-out patients at matched column count, with the gap concentrated on small-extent lesions.

The strongest existing vision method is the Medical Slice Transformer. It trains an attention model across DINOv2 slice features and fine-tunes the encoder, and it reports AUC 0.85 on MRNet (against 0.69 for a 3D ResNet50), 0.95 on LIDC-IDRI, and 0.94 on Duke-Breast-Cancer-MRI. Its authors also argue that encoder fine-tuning is needed for radiologic textures, which is the direct threat to a frozen pipeline. We will retrain it on our patient-level splits at every training-set size, with a hyperparameter search budget matched in GPU-hours to the total cost of our column selection plus TabPFN bagging. No reported result was found on Lacuna for this model at reduced training-set sizes. Beating it means higher AUC at 50 to 500 training patients. At full data we expect it to win, and we will report the crossover point rather than hide it.

For imaging plus tabular data, the strongest reported method is HyperFusion. It uses clinical variables to generate weights inside a 3D CNN and reports ADNI three-class balanced accuracy of 0.673, against 0.658 for the DAFT conditioning method and 0.579 image-only. It conditions on only 9 clinical variables, which is the range our selection-based rows are meant to exceed. We will rerun it on our ADNI splits with the same clinical variables. Beating it means balanced accuracy above its rerun value, especially at reduced cohort sizes.

The key ablation removes the patient-relative median, replacing $m$ with the cohort median. It also swaps mutual-information selection for PCA at fixed width. These two changes isolate the two ingredients we claim matter.

Why not just…?

Why not just mean-pool the slice embeddings, as Images as Tables does for 2D images? This is the first baseline, and the dilution experiment below targets it directly. Averaging $S$ slices shrinks a single-slice deviation by a factor of $S$. CARVE's evidence that diagnostic content can sit on one or two slices suggests this loss is real. However, the file contains no measurement of mean pooling's failure on small lesions, so this remains a hypothesis to test. If mean-pooled rows match residual rows even on the smallest LIDC-IDRI nodules, the core claim fails.

Why not just fine-tune the Medical Slice Transformer and concatenate the clinical variables before its output layer? With enough labels this may simply win. Images as Tables itself found that a specialized discriminator overtakes TabPFN by 7.4% once data exceeds about 5,000 samples. HyperFusion reports that late concatenation tends to underuse tabular data, but that evidence comes from a 3D CNN, not a slice transformer. We therefore run the learning-curve experiment and report the training-set size at which fine-tuning overtakes in-context prediction. Our claim is limited to the regime below that point.

Why not just train CatBoost on the same residual columns? This is the most serious alternative, because boosted trees handle noisy columns more gracefully than TabPFN (Scaling TabPFN). If CatBoost on residual rows matches TabPFN, the in-context learner adds nothing, and the honest contribution is the volume-to-row representation. Every main result will be reported under TabPFN, CatBoost, and logistic regression.

Why not just learn the slice weighting with attention-based multiple-instance pooling? A learned pooling head is the trainable version of our hand-built statistic. With few hundred patients it may overfit, but that is an empirical question. We include it at matched tuning budget. If it beats fixed residual statistics in the low-data range, we would adopt learned pooling and keep only the in-context classification step.

Proposed Experiments

The cohorts are LIDC-IDRI chest CT, MRNet knee MRI, Duke-Breast-Cancer-MRI, and ADNI brain MRI with its clinical variables. All use patient-level splits, repeated over five seeds, with training sets of 50, 100, 250, 500, and all available patients. The metric is AUC, plus balanced accuracy on ADNI's three-class task.

The first decisive check stratifies LIDC-IDRI test nodules by the number of slices they span, computed from the dataset's annotations. The prediction is that the gap between residual rows and mean-pooled rows widens as nodule extent shrinks. A flat gap would falsify the mechanism even if average AUC improves.

The second check is a dilution test. We pad each test and context volume with additional slices drawn from that patient's own lesion-free region and track AUC as padding grows. Mean-pooled rows should decline steadily, while residual rows should stay near flat.

The third check tests the role of the patient-relative median. If cohort-median residuals perform equally well, the patient-relative framing is unnecessary.

The fourth check addresses the frozen-versus-fine-tuned tension. We rerun our pipeline with the slice transformer's fine-tuned encoder substituted for the frozen one. This tells us whether residual rows still help once the features are adapted to radiologic texture.

Finally, on ADNI, we test whether adding clinical columns to residual rows closes the gap to HyperFusion at small cohort sizes. That would show that in-context prediction over a joint row can replace learned cross-modal conditioning when labels are scarce.