Sparse Spatial Probes for Frozen 3D CT Encoders in Small Cohorts

Large pretrained encoders for 3D CT scans are increasingly used in a simple way. Each scan is compressed into one vector by averaging over the whole volume, and a linear classifier is trained on that vector for a new clinical question. This works for large, bright abnormalities. It fails for small, faint lesions, because averaging over millions of voxels dilutes a lesion that occupies a few hundred. This is the main finding of Big, Bright, or Invisible, a frozen-feature benchmark of ten 3D CT encoders. The obvious fix is to keep the encoder's local patch embeddings instead of averaging them. That fix runs into a second problem. Clinical cohorts for new questions often have 20 to 100 labeled patients, and a classifier over thousands of unpooled local features memorizes noise at that scale.

We propose to resolve both problems with one constraint borrowed from classical statistics: a structured sparsity penalty, meaning a training penalty that forces most weights to exactly zero while encouraging the surviving weights to form spatially contiguous blobs. Each scan is aligned to a common anatomical grid. The frozen encoder's patch embeddings are compressed to a few components per grid cell. A linear model is then trained with three penalties: one that switches whole grid cells off, one that makes neighbouring cells agree, and one that zeroes individual channels. The claim is that this "sparse spatial probe" recovers focal, low-contrast findings that pooled probes miss. It should do so at sample sizes where flexible local aggregators such as attention pooling overfit, and it should return a readable map of where in the body the evidence came from.

Core Claim

Whole-volume pooling loses volumetric context in a specific way: it discards where the signal is, not only how it continues across slices. In small cohorts, location is the cheapest prior available. A finding in the right lower lobe sits in roughly the same atlas region across patients, so a model that learns weights for a small, contiguous set of regions needs far fewer labels than one that must learn, from scratch, which patches to attend to. We expect three things. First, a graph-fused, group-sparse linear probe on atlas-aligned patch embeddings should beat a pooled ridge probe on small, low-contrast findings, with the largest gains at the smallest sample sizes. Second, it should match or beat attention-based pooling up to about 100 labeled patients. Third, the cells it selects should overlap the documented location of the finding. If attention pooling or simple max pooling matches it at n ≤ 50, the structured-sparsity claim is wrong, and we will report that.

The prior work this builds on plays three different roles. Big, Bright, or Invisible motivates the proposal. It shows that physical footprint and contrast govern what frozen 3D encoders can detect, and that linear probes extract more signal than zero-shot prompting. However, it evaluates only globally pooled probes and k-nearest-neighbour classifiers, so it never tests unpooled local features or sparse probe weights. Fused lasso for feature selection using structural information supplies the method family: a fused lasso over a feature graph that selects related features together. It was demonstrated on 2D images, microarrays and speech, and its authors note that the linear systems become expensive when the feature count exceeds roughly 10^5. We apply the idea to a 3D anatomical grid. By compressing channels first, we keep the problem near 10^4 features, where the cost stops being a concern. CARVE is the closest competitor in spirit. It selects which 3D tokens a multimodal language model sees, using depth-adaptive windows, quadtree anchors and residual retrieval, and keeps 98.1% of full-token performance at a 20% keep ratio. But its selection is a fixed heuristic, independent of labels, and its authors name a learnable, task-adaptive selector as future work. Our selection is driven by the prediction loss and is fit on tens of patients rather than designed for large question-answering benchmarks.

Research Program

The pipeline has four stages. First, each CT volume is affinely registered to a thoracic atlas, which is a standard reference anatomy. The atlas is divided into R cells defined in millimetres, so scans with thick and thin slices map to the same physical grid. This is how through-plane anisotropy (slice spacing much coarser than in-plane resolution) is handled. Second, a frozen 3D encoder produces patch embeddings, which are resampled into the cells. Third, principal component analysis, fit once on unlabeled scans (for example the CT-RATE training split), reduces each cell's embedding to k components. With R around 512 and k around 16, each patient is described by roughly 8,000 features. Fourth, we fit a weight matrix B, with one k-dimensional weight block $B_r$ per cell, by minimizing

$$\frac{1}{n}\sum_{i=1}^{n}\ell\Big(y_i,\ \sum_{r=1}^{R}\langle B_r, x_{i,r}\rangle + b\Big) + \lambda\Big[\alpha_1\sum_{r}\|B_r\|_2 + \alpha_2\!\!\sum_{(r,r')\in E}\!\omega_{rr'}\|B_r-B_{r'}\|_2 + \alpha_3\|B\|_1\Big].$$

Here $x_{i,r}$ is patient i's compressed embedding in cell r, $y_i$ is the label, $\ell$ is the logistic loss (or the Cox partial likelihood for survival), and b is an intercept. E is the set of adjacent cell pairs, and $\omega_{rr'}$ is a weight that decays with the physical distance between cell centres, so cells that are closer together in millimetres are pulled harder toward the same weights. The first penalty switches whole cells off, the second makes neighbouring cells agree, and the third zeroes individual components.

Tuning several penalty strengths is fragile when validation sets contain a handful of patients. To avoid that, we fix the mixing ratios $\alpha_1, \alpha_2, \alpha_3$ once on a separate development task and tune only the overall strength λ by nested cross-validation. We add stability selection, which refits the model on many half-samples and keeps only cells chosen in a large fraction of refits, to report which regions are reliably selected. The model is fit by the alternating direction method of multipliers, a standard solver for problems with several non-smooth penalties. At this size each fit takes seconds, which is what makes nested resampling at small n affordable.

The analysis has three parts. We measure where gains appear as a function of finding size and contrast, using the size and contrast stratification axes from the benchmark paper. We measure how gains change as the number of labeled patients shrinks. And we check whether the selected regions agree with report-derived location labels such as lobe and laterality.

Baselines

The naive reference point is the protocol of Big, Bright, or Invisible: a whole-volume mean-pooled embedding with an L2-regularized (ridge) logistic probe. Our method must beat this baseline to justify itself, and it is the protocol the community actually uses. It runs on the same encoders, the same training subsamples and the same nested-cross-validation budget for its single penalty strength. That paper reports probe performance only graphically, so there is no reported result found on Lacuna for per-finding linear-probe AUROC on CT-RATE or RAD-ChestCT. We will therefore reproduce this baseline ourselves rather than quote it.

The strongest learned competitor is attention-based multiple-instance pooling. A small network scores each patch embedding, and the classifier sees the score-weighted average. This is the standard way to localize evidence without registration, and at moderate n it is the most likely method to beat ours. To keep the comparison fair, it receives the same PCA-compressed patch embeddings and the same number of hyperparameter configurations. Its architecture is kept small (one hidden layer) so that its capacity is not a strawman. A second strong but simple competitor is max pooling or top-k pooling over the same patches, followed by a ridge probe. It captures focal signal with no learned spatial structure at all.

The strongest end-to-end competitor is partial fine-tuning of the encoder's last block plus a linear head. This is the transfer strategy used by the voxel-level NSCLC radiomics study, which reports a C-index of 0.73 (95% CI 0.63–0.82) on an external immunotherapy test set of 150 patients after fine-tuning on only 37. If we can obtain that cohort, this C-index is the number to beat on survival. If not, we will run the fine-tuning baseline ourselves on the thoracic finding tasks with matched compute.

Ablations isolate the key ingredient. We drop the fusion penalty, which leaves a plain group-sparse probe. We drop the group and fusion penalties, which leaves an elastic net on unpooled features. Finally, we keep the full penalty but randomly permute each patient's cell positions. This destroys spatial correspondence while keeping the feature count and sparsity level, and it directly tests whether anatomical location, rather than sparsity alone, drives any gain. Beating a baseline means a higher mean AUROC (or C-index) across repeated subsamples, with a paired bootstrap confidence interval that excludes zero, on findings in the small or low-contrast strata.

Why not just…?

Why not just use attention pooling over the patch embeddings? This is the real threat. Attention pooling learns which patches matter from the labels alone, whereas our probe also assumes that relevant patches sit in a consistent, contiguous anatomical region. The literature in the file offers no direct evidence on attention pooling at n ≤ 50 on 3D CT, so we cannot argue it away. The sample-size sweep settles it. If attention pooling matches the sparse probe at 20 and 50 labeled patients, our central claim fails. We expect it to catch up at a few hundred patients, and the crossover point is itself a useful result to report.

Why not just fine-tune the encoder, as the radiomics transfer work did? The NSCLC study made fine-tuning on 37 patients work only after pre-training on a related 271-patient chemotherapy cohort. That kind of source cohort is rarely available for a new finding. The file also notes that directly fine-tuning large 3D models on small cohorts degrades their representations. Our partial fine-tuning baseline tests whether this holds without a matched source cohort.

Why not just use ridge or an elastic net, since L2 already prevents overfitting? The fused lasso paper reports that unstructured Lasso, Ridge and Elastic Net lose to structure-aware penalties on high-dimensional benchmarks (InFusedLasso). The fMRI connectivity literature reports a related warning: small-cohort accuracy collapses across sites when models latch onto spread-out spurious signal (Du et al. review). Neither result is on CT. The elastic-net and permuted-position ablations test this question directly. If the elastic net matches the full model, the spatial structure is unnecessary, and we would say so.

Why not just slice the volume and use a strong 2D model? Independent 2D slice predictions are known to lose depth continuity, dropping a structure on one slice that they caught on the next (pancreas CT segmentation). Uniform slice subsampling risks missing the one or two slices that contain a focal lesion (CARVE). We include a 2D slice-embedding baseline with max-over-slices aggregation for completeness, but we expect it to trail the 3D encoders.

Proposed Experiments

The main testbed is the pair of public thoracic CT cohorts used in the frozen-feature benchmark: CT-RATE (1,564 evaluation scans) and RAD-ChestCT (2,284 scans), each with report-derived multi-label findings. We use publicly released encoders among those the benchmark found strongest (FlexiCT, COLIPRI and Pillar-0). For each finding, we draw balanced training sets of n = 20, 50, 100 and 200 patients, repeat each draw 20 times, and evaluate on the full held-out split. We also train on CT-RATE and test on RAD-ChestCT to measure the cross-site drop that the connectivity review warns about. Results are stratified by the size and contrast of the finding.

The decisive check is a specific interaction. The sparse probe's AUROC gain over the pooled ridge probe should be largest for small, low-contrast findings and should shrink for large, bright findings such as support devices. Over attention pooling, the gain should be positive at n ≤ 50 and should shrink as n grows. If gains are uniform across finding types, the mechanism is generic regularization, not recovery of focal signal. If the permuted-position ablation performs as well as the full model, anatomical location contributes nothing.

The second check is localization. For findings whose reports give a lobe or a side, we measure how much of the stable selected region falls in the documented region and compare it with the attention maps from the attention-pooling baseline. A secondary survival experiment uses the RTOG 0617 cohort with the Cox loss, and the immunotherapy cohort if access allows, against the 0.73 C-index reported by the cross-dataset radiomics study. Here the question is narrower: can a frozen encoder with a sparse spatial probe approach a fine-tuned, source-pretrained model without any source cohort?