Large pretrained 3D CT encoders are becoming the default way to build clinical classifiers when only a few dozen labeled scans exist. The encoder stays fixed, and a small model is fitted on its output vectors. A recent benchmark of ten such frozen encoders found a hard limit. Large or bright findings, such as implanted devices or fluid collections, are easy to read out. Small, faint findings, such as subsolid lung nodules, largely disappear. The authors attribute the failure to the physical size and contrast of the finding, compressed by averaging the whole volume into one vector, rather than to encoder architecture (Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models). This note proposes keeping the encoder frozen but not averaging its output. We treat the encoder's spatial grid of patch features ("tokens") as a graph in which each token connects to its 3D neighbors. A small graph neural network then passes messages along this grid before a final pooling step. A graph neural network (GNN) is a model that updates each node's features from its neighbors.
The specific bet concerns direction. A faint nodule and a vessel seen in cross-section can look alike within a single token. They differ in how the token relates to its neighbors in different directions. A vessel continues along one axis, while a nodule differs from its surroundings in all directions. Ordinary "isotropic" message passing sums or averages neighbors without regard to direction, so it cannot tell these two patterns apart. "Anisotropic" message passing weights each neighbor by its feature and by its 3D offset, and it can. We claim that a parameter-matched anisotropic head on frozen tokens can recover small focal findings at label budgets of 25 to 200 scans. We also claim that the gain over an isotropic head should concentrate on findings that are easily confused with tubular anatomy.
Core Claim and How It Differs From Prior Work
The claim has three layers, and the experiments are built to separate them. First, part of the "invisible" regime in Big, Bright, or Invisible comes from pooling rather than from the encoder, so some token-level readout should already help. Second, comparing each token with its neighbors adds information beyond reading tokens one at a time. Third, that comparison must be direction-aware to separate blobs from tubes.
If only the first layer holds, the right answer is attention pooling over tokens, and no graph is needed. If the first two hold but not the third, a fixed hand-crafted contrast feature or an isotropic GNN suffices. We state this plainly because the simpler outcomes are live possibilities.
The Chevli benchmark motivates the work. It reads out global embeddings, independent tokens, k-nearest-neighbor classifiers, and text prompts, but it never relates tokens to one another spatially. The anisotropic mesh literature supplies the technique and shows that direction matters:
- GEM-CNN builds direction-sensitive kernels on curved surfaces using local coordinate frames and parallel transport. It cuts Embedded MNIST error from 19.8% with an isotropic GCN to 0.60% (Gauge Equivariant Mesh CNNs).
- On artery-wall shear stress, anisotropic FeaSt and GEM-CNN beat isotropic GraphSAGE (Mesh convolutional neural networks for wall shear stress estimation in 3D artery models).
- The follow-up GEM-GCN reaches 0.4% normalized mean absolute error (Mesh Neural Networks for SE(3)-Equivariant Hemodynamics Estimation on the Artery Wall).
All three train from scratch on thousands of synthetic simulations, using raw geometric inputs. We instead put anisotropy on top of frozen learned features, with tens of labels.
We also need none of their gauge machinery. A CT volume comes with a canonical patient coordinate frame (left–right, front–back, head–foot). Offsets between tokens therefore have a well-defined direction, and the precomputation overhead and approximate equivariance that GEM-CNN accepts do not arise. The edge-gating design comes from GatedGCN, which Benchmarking Graph Neural Networks found beats isotropic GNNs under fixed parameter budgets.
Research Program
For each scan, we run a frozen encoder and keep its unpooled token grid. We use only encoders from the benchmarked set, such as CT-CLIP, COLIPRI or FlexiCT, that expose spatial tokens; any that do not are excluded rather than adapted. Each token becomes a node with feature $h_i$. Each node connects to its 26 grid neighbors, and edge $(i,j)$ carries $e_{ij}$, an encoding of the unit offset direction from $i$ to $j$ in patient coordinates. One layer of the head computes
$$ h_i' = h_i + \frac{\sum_{j} \eta_{ij} \odot V(h_j - h_i)}{\sum_j \eta_{ij} + \epsilon}, \qquad \eta_{ij} = \sigma\big(A h_i + B h_j + C e_{ij}\big). $$
Here $\eta_{ij}$ is a per-channel gate in $(0,1)$ produced by the sigmoid $\sigma$. $A$, $B$, $C$ and $V$ are learned linear maps, $\odot$ is elementwise product, and $\epsilon$ prevents division by zero.
This is GatedGCN with two deliberate changes. Messages carry the difference $h_j - h_i$, so the head computes local contrast rather than local averages. Gates see direction, so contrast can be weighted differently along a vessel's axis than across it. Two or three layers are followed by attention pooling over nodes and a logistic output. The whole head is held to roughly 100k parameters, following the fixed-budget protocol of Benchmarking Graph Neural Networks.
The memory cost of storing one value per edge is a known concern (Do We Need Anisotropic Graph Neural Networks?). Here it is small: a token grid of a few thousand nodes with 26 neighbors each is far smaller than the citation graphs where that cost caused out-of-memory failures. We will report peak memory regardless.
The small-sample regime is the operating constraint, not an afterthought. Unconstrained models on cohorts under 100 patients latch onto scanner and site artifacts (Towards AI-driven Next Generation Personalized Healthcare and Well-being). Freezing the encoder keeps training cheap and stable; Zhang et al. report up to 21× faster convergence than full fine-tuning (Zhang et al.). It also makes the comparison clean, because every method sees identical features.
Baselines
The naive reference point is the protocol of Big, Bright, or Invisible: a linear probe on the globally pooled embedding of the same frozen encoder. It is also the strongest existing method on exactly this question. It is fitted with the same labeled scans, the same splits, and an L2-regularization sweep of the same size as the head's tuning budget. The research file gives no per-finding AUROC (area under the ROC curve) values: no reported result found on Lacuna for linear-probe AUROC on CT-RATE or RAD-ChestCT focal findings. We will therefore reproduce it ourselves, and treat matching the paper's qualitative size-and-contrast trend as a sanity check on our pipeline.
The second baseline is token-level attention pooling with no message passing. It fits a scoring network that weights each token and sums them, at the same parameter budget. This is the strongest graph-free competitor: if pooling is the whole problem, it should close the gap.
Two further baselines isolate what the learned graph adds over cheap, fixed features computed before attention pooling. The isotropic version appends to each token its difference from the mean of its neighbors. The directional version appends the 13 axis-wise neighbor differences.
Two learned graph heads isolate the anisotropy itself:
- Isotropic GNN: GraphSAGE with mean aggregation, using the same difference messages.
- Gated without direction: the proposed head with the direction term $C e_{ij}$ removed. This is the key ablation.
Do We Need Anisotropic Graph Neural Networks? argues that adaptive node-level filtering (Efficient Graph Convolution, EGC) can match anisotropic models at lower cost, so EGC is included at matched budget.
As an upper reference, we fine-tune the encoder's top blocks. In histopathology, fine-tuning upper blocks beat a frozen-feature SVM, 56.98% versus 50.54% accuracy (Convolutional Neural Networks for Histopathology Image Classification), so this comparison is not a formality.
Every head gets the same random-search budget of 30 configurations, chosen by inner cross-validation on the training labels only, and 10 resampled label subsets per budget. "Beating" a baseline means a higher mean AUROC on the focal-finding group, with a paired bootstrap 95% interval excluding zero. This must hold for at least two encoders at label budgets of 100 or fewer, with no loss beyond 0.01 AUROC on large findings.
Why not just…?
Why not just use a linear probe on the frozen global embedding? It is the cheapest option and immune to overfitting the graph. However, the Chevli benchmark shows its failure is specifically on small, faint findings, the target here. The comparison stays in the proposal as the floor, and it can still win on large findings, which we report rather than hide.
Why not just pool tokens with attention and skip the graph? This is the most serious threat. If tokens already encode a nodule distinctly, attention pooling will find it, and message passing adds parameters for nothing. Our guard is the "encoder ceiling" measurement described below. If the attention-pooling baseline matches the anisotropic head on focal findings, we will report that pooling, not relational structure, was the bottleneck, and drop the graph.
Why not just add a fixed contrast feature, or use an isotropic GNN or EGC? If the needed signal is "this token differs from its surroundings," a fixed neighbor-difference feature delivers it with no learning. The tube-versus-blob test below is where these alternatives should fail. An isotropic difference cannot separate a vessel crossing a token from a nodule of equal volume. If the fixed 13-direction feature matches the learned anisotropic head, the honest conclusion is that direction matters but learning the gates does not. The ladder of baselines is designed to report exactly that.
Why not just fine-tune the encoder? It may well win when labels are plentiful. Below 100 scans, though, it risks the site-artifact memorization documented by Bria et al. and costs far more compute. We report the crossover label budget at which fine-tuning overtakes the frozen anisotropic head, and the cross-dataset drop for each method.
Proposed Experiments and Decisive Checks
Data come from CT-RATE and RAD-ChestCT, the public cohorts in the Chevli benchmark. Findings are grouped by physical extent and contrast following that paper: devices and effusions are "large"; nodules and small focal opacities are "focal." Label budgets are 25, 50, 100 and 200 positive-plus-matched-negative scans. We train on one dataset and test both in-domain and on the other dataset, to probe the cross-site degradation that small-cohort models suffer.
The first decisive check is the encoder ceiling. On scans with localized nodule annotations, or with synthetic inserts as described next, we test whether a token-level probe can separate lesion-containing tokens from their own neighbors at all. If it cannot, no head on frozen tokens can help, and the project's answer is "use a finer encoder." That is a useful negative result.
The second check is controlled insertion, in the spirit of Embedded MNIST. We insert synthetic low-contrast spheres and synthetic tubes of equal voxel volume into normal lung regions across a grid of diameters and contrasts, then pass the volumes through the frozen encoder. We measure two things: the size–contrast boundary at which each head's detection AUROC falls to 0.6, and the AUROC for telling spheres from tubes. The proposal's prediction can fail cleanly. Removing $C e_{ij}$ should leave sphere detection roughly intact but markedly hurt sphere-versus-tube discrimination. On real data, the anisotropic-minus-isotropic gain should be larger for nodules than for focal findings without tubular mimics.
If these patterns do not appear, the directional story is wrong, whatever the headline AUROC says.
References
- Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models
- Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs
- Mesh Convolutional Neural Networks for Wall Shear Stress Estimation in 3D Artery Models
- Mesh neural networks for SE(3)-equivariant hemodynamics estimation on the artery wall
- Benchmarking Graph Neural Networks
- Do We Need Anisotropic Graph Neural Networks?
- Towards AI-driven Next Generation Personalized Healthcare and Well-being
- Efficient Pre-Trained Features and Recurrent Pseudo-Labeling in Unsupervised Domain Adaptation
- Convolutional neural networks for histopathology image classification: Training vs. Using pre-trained networks