Shared 3D Anatomical Parts for Few-Shot Organ Segmentation

When a segmentation model meets a new organ or vessel with only one labeled scan, it usually treats that structure as a single indivisible target. Yet much of the new structure has often been seen before as part of other organs. A renal artery looks a lot like a hepatic vessel: a bright tube with a wall and occasional branch points. The gallbladder and the bladder are both fluid-filled sacs with a thin membrane. This proposal builds a segmentation model for CT and MRI volumes that learns a small shared vocabulary of three-dimensional "anatomical parts" across many organs. Each organ, including one never seen in training, is expressed as a weighted combination of those parts and of their typical 3D arrangements. The goal is to segment a new structure from one to five support volumes by recombining parts the model already knows, instead of matching a whole-organ template.

The bet is testable, and it can fail in an informative way. If shared parts are what carry the transfer, the benefit should be largest when the new structure is built from parts that appear in the training organs. The benefit should shrink, measurably, when we deliberately remove every organ containing those parts from training. Prior few-shot volumetric methods cannot make this prediction because they have no explicit parts to remove. This lets the proposal tell genuine compositional transfer apart from "a bigger or better-pretrained model helps."

Core Claim

The claim has three parts. First, a dictionary of part-like feature prototypes, learned jointly across partially labeled 3D CT datasets, gives better 1- and 5-shot segmentation of unseen structures than whole-organ support prototypes. Second, the dictionary must be paired with 3D spatial layout patterns over those parts, not 2D per-slice layouts. Third, the size of the gain is predicted by how well the frozen dictionary can reconstruct the new structure's support features. The first part is the method claim. The third part is the scientific claim that the method makes testable.

Relation to Prior Work

Two bodies of work motivate the idea, and each lacks what the other has.

2D part sharing without volumes. COMPAS builds a component dictionary of part prototypes, found by unsupervised clustering, together with a dictionary of spatial activation layouts inside a 2D convolutional network. With a frozen backbone and a simple logistic-regression classifier, it reports 65.74% 1-shot and 83.03% 5-shot accuracy on miniImageNet. We reuse its two-dictionary design almost directly. COMPAS works on 2D natural images, and its own stated limitations are that 2D layouts break under deformation and that the dictionary fails when test classes come from a different domain. We move both dictionaries into 3D, where layouts are not distorted by projection. We also turn the domain-gap limitation into a controlled experiment rather than a caveat.

Volumetric few-shot and multi-organ methods without parts. These methods supply the application regime and the baselines, and each stops short of part sharing in a specific way:

  • MSFSeg segments unseen organs from a few support slices. It fuses support features through intersection, union, attention, and averaging "surrogates," but it treats each support structure as one whole. It reports 77.14 Dice at 1-shot and 83.27 at 5-shot on unseen Abdomen-CT organs.
  • MetaMedSeg meta-learns a starting point for a 2D U-Net, using tasks sampled from within a single volume. It has no explicit parts and no native 3D features.
  • CCQ and DoDNet train on partially labeled CT collections, where each dataset annotates only some organs. CCQ shares context between whole-organ queries and reaches 76.41% average Dice on the MOTS benchmark. It shares nothing below the organ level and cannot add a new class without retraining.

We borrow their partial-label training setup but share parameters at the sub-organ level.

3D part work outside medicine constrains the design. Learning Shape Abstractions by Assembling Volumetric Primitives assembles shapes from cuboids. Its own authors note that rigid parametric shapes cannot represent curved organic structures, so our parts are learned feature prototypes rather than geometric solids. VoxAttention shows that relational assembly beats per-part coordinate regression on ShapeNet chairs (78.4% versus 71.8% shape mIoU), but it learns parts per category on canonically aligned CAD models. P3-SAM segments parts natively in 3D but works only on surface point clouds, with no cross-category semantics. That rules it out for intensity volumes where the useful evidence lies inside the organ.

Choice of backbone. The AMOS benchmark found that a plain 3D U-Net (89.04% mDice on CT) beat Swin-UNETR (86.32%). We therefore use a 3D U-Net encoder rather than a transformer, which keeps any gain attributable to the dictionary rather than to the backbone.

Research Program

The model has three components.

Encoder. A 3D U-Net encoder produces a feature vector $f(v)$ at each voxel $v$ on a coarse grid.

Component dictionary. A learned set of $K$ unit-norm atoms $D=\{d_1,\dots,d_K\}$, with $K$ between 64 and 256, turns each voxel feature into soft part activations $a_k(v)$. These are a softmax over cosine similarities between $f(v)$ and the atoms.

Layout dictionary. A set of $M$ small 3D kernels $S_m$ captures recurring spatial arrangements of part activations, such as a tube activation flanked by a wall activation along a line. Each kernel is convolved over the stack of activation maps.

Each class $c$ is then a readout over parts and layouts. Its voxel score is

$$s_c(v)=\sum_{k=1}^{K} w_{c,k}\,a_k(v)+\sum_{m=1}^{M} u_{c,m}\,(S_m * a)(v),$$

where $w_{c,k}$ and $u_{c,m}$ are class-specific weights and $(S_m * a)$ is the 3D convolution of layout $m$ with the part-activation maps. Only these weights are class-specific. Adding a class costs $K+M$ numbers rather than a new decoder, which is the sub-linear scaling that Supervised Dictionary Learning and compositional-hierarchy work argue for.

Base training. We train on the union of partially labeled datasets (MOTS and the AMOS CT training split, with held-out classes removed). For each class, the segmentation loss is computed only on scans where that class is annotated. Other voxels are ignored, following the masking used in 3D U-Net, so unlabeled organs are not pushed toward background. Two auxiliary terms prevent degenerate dictionaries: a coverage term that rewards reconstructing $f(v)$ from a few atoms, and a decorrelation term between atoms. Atoms are initialized by k-means over encoder features, as in COMPAS.

Novel classes. For a new structure, the encoder and both dictionaries stay frozen. We fit only $w_c$ and $u_c$ on the one to five support volumes, using logistic regression with a fixed regularization strength chosen on validation base classes. As a lightweight diagnostic, we compute each novel class's reconstruction residual $r_c$: the mean error of the best sparse reconstruction of its support foreground features from $D$. The program then has three steps: build the model, check that the atoms are anatomically meaningful, and test whether $r_c$ predicts transfer.

Baselines

Naive reference. A 3D U-Net trained from random initialization on the support volumes alone, with standard augmentation. It shows how much any form of prior knowledge matters.

Strongest existing few-shot method. MSFSeg, run from its published design on our exact splits, with the same support volumes and the same base-training data. On Abdomen-CT setting 2 (classes unseen in training), MSFSeg reports 77.14 Dice at 1-shot and 83.27 at 5-shot. We first reproduce it on that benchmark to check our implementation. Beating it means a higher mean Dice than our reproduced MSFSeg on the same unseen classes, averaged over at least five support draws, with paired confidence intervals that exclude zero. For context, no reported few-shot result on CHAOS-MRI was found on Lacuna for MSFSeg, so there we compare only against our own runs.

Fine-tuned pretrained 3D U-Net. The same encoder, pretrained on the same base data with a conventional multi-head decoder, then fine-tuned end to end on the support volumes. Every method gets an equal hyperparameter search of 20 configurations on base-class validation tasks. This is the most likely real competitor.

Two ablations isolate the key ingredients. The first removes sharing. The dictionary is replaced by per-class prototypes, made by k-means clustering of each novel class's own support features into $K$ clusters, with the same readout. This tests whether part-like multi-prototype matching alone explains any gain. The second removes volumetric layout. The layout kernels $S_m$ are restricted to single-slice 2D kernels, giving a COMPAS-style model applied per slice. We also report a dictionary-without-layouts variant.

Large foundation model. SegVol reports 0.8593 average Dice on AMOS22. It was trained on 25 public datasets that almost certainly include our "novel" organs, so its numbers are reported only as a contaminated upper reference, not as a fair competitor.

Why not just…?

Why not just fine-tune a well-pretrained 3D U-Net on the support scans? This is a serious threat. AMOS shows plain 3D U-Nets are hard to beat, and pretraining on many organs may already store tube and sac detectors in its filters. The fine-tuned pretrained 3D U-Net baseline uses the same encoder and data, so it directly measures what the explicit dictionary adds beyond pretrained filters. If fine-tuning matches us at 5-shot, the method's claim narrows to the 1-shot regime and to interpretability, and we will say so.

Why not just use several prototypes per class, clustered from the support set, as a stronger MSFSeg? If cross-category sharing is irrelevant, this per-class-prototype ablation will match the shared dictionary, because it also represents a structure as several pieces. The decisive difference is where the pieces come from. With one support volume, prototypes clustered from that volume alone are fit to one scan's noise. Shared atoms are estimated from thousands of voxels across many organs. We expect the gap to be largest at 1-shot and for thin structures with few foreground voxels.

Why not just prompt a foundation model like SegVol? SegVol's authors note that it maps whole organ names through text embeddings and cannot parse structural descriptions. Fair few-shot evaluation is also impossible when its training data contains the test organs. We therefore cannot claim to beat it on familiar anatomy. Our claim concerns structures absent from base training, where its text space has no entry to look up.

Why not just apply a 2D part dictionary slice by slice? The 2D-layout ablation answers this. Evidence from 3D U-Net (0.863 versus 0.796 IoU for 3D versus 2D) and from 3D Deeply Supervised Network suggests that cross-slice context matters most for oblique tubes and low-contrast boundaries. We therefore expect the gap to concentrate on vessels rather than on large compact organs.

Proposed Experiments

Splits. Novel classes are drawn from AMOS's 15 organs plus the hepatic vessel class in MOTS, in two kinds of split:

  • Overlap split. We hold out the aorta and inferior vena cava while keeping hepatic vessels in base training, and hold out the gallbladder while keeping the bladder.
  • Starved split. We hold out the same targets but also remove every tubular or sac-like structure from base training.

Decisive check. The main test is a difference-in-differences. The gain of the shared dictionary over the per-class-prototype ablation should be clearly positive on the overlap split and should shrink substantially on the starved split. Across all novel classes, that gain should correlate negatively with the reconstruction residual $r_c$. If the gain is the same in both splits, the improvement does not come from sharing parts, and the central claim is refuted even if mean Dice goes up.

Other evaluations.

  • Benchmark checks. Abdomen-CT setting 2, to compare against MSFSeg's published numbers, and CHAOS-MRI for cross-modality transfer.
  • Metrics. Dice, Hausdorff distance on MOTS-derived classes, and a slice-to-slice consistency score, defined as the Dice between masks on adjacent slices relative to ground truth.
  • Scaling. Class-specific parameter counts as the number of base classes grows.
  • Atom inspection. A qualitative check that individual atoms fire on recognizable sub-structures across different organs.