Paying for Parts: Instance-Adaptive Part Discovery in 3D Volumes

Most systems that break a 3D object into parts make the user choose the number of parts, $K$, before training. Every chair, airplane, or brain scan then gets exactly $K$ pieces. A simple instance ends up split into arbitrary fragments, and a complex one has distinct structures merged together. The systems that do let $K$ vary from one instance to the next work on surface points or rendered 2D views. They cannot see the interior of a solid volume, which is the only place where, for example, a tumor's nested sub-regions or an organ boundary exist. Meanwhile, networks that read volumes natively, using 3D filters that look across all three axes at once, either predict a fixed list of named classes or require a human prompt for every structure.

We propose replacing the fixed part count with a fixed price per part. A 3D network reads the volume and recursively proposes to split regions in two. It keeps a split only if the improvement in reconstructing the voxels inside that region is larger than a single global cost $\lambda$. Because the price is shared across all instances, the number of parts can differ per instance: a plain stool stops after three splits, an armchair continues to eight, and a scan showing more organs gets more parts. The result is a feedforward, trainable analogue of the nonparametric Bayesian part models, with no upper bound on the part count and a part hierarchy as a by-product.

Core Claim

The claim is narrow and testable. If one trades a global part count $K$ for a global per-part cost $\lambda$, and judges each part by how much volumetric content it explains rather than by how compact it is in feature space, then at a single $\lambda$ the number of discovered parts will track the true per-instance part count. A fixed-$K$ model cannot do this by construction. We also predict that part discovery will be more accurate when the split decisions use native 3D context than when they use 2D slices or surface points.

Formally, a decomposition of a volume $V$ into leaf regions $R_1,\dots,R_M$ is scored as follows:

$$ \mathcal{L}(V) = \sum_{m=1}^{M} \mathrm{err}\big(V_{R_m}, \hat{V}_{R_m}\big) + \lambda M, $$

Here $\hat{V}_{R_m}$ is the reconstruction of the voxels in region $R_m$ from a single low-dimensional part code, and $\mathrm{err}$ is the reconstruction loss (occupancy cross-entropy for shapes, intensity error for scans). The leaf count $M$ is chosen per instance. This is the hard-assignment limit of a Dirichlet-process mixture, the same "small-variance" argument that turns a Chinese Restaurant Process prior into a k-means cost plus $\lambda$ per cluster. A split of region $R$ into children $R', R''$ is accepted when $\mathrm{err}(R) - \mathrm{err}(R') - \mathrm{err}(R'') > \lambda$.

Relation to Prior Work

Four lines of work motivate the gap or supply methods we reuse.

Primitive-Based Shape Abstraction via Nonparametric Bayesian Inference infers the number of superquadric primitives with a Chinese Restaurant Process prior, using optimization-based Gibbs sampling and a merging step. It supplies our objective: we amortize its "add a part only if it pays" logic into a single forward pass. Unlike that work, we operate on dense volumes rather than isolated points, and we avoid per-instance iterative fitting.

Rethinking 3D Shape Generation: Diffusion over Superquadrics handles variable cardinality with a 128-slot buffer and a per-slot existence score. It needs an imposed canonical ordering of slots, and its upper bound is hard-coded. Recursive splitting removes both the ordering problem and the cap.

Neural Parts shows that giving each part more expressive power reduces how many parts are needed: it reports 67.3% IoU on D-FAUST with 5 primitives, against 62% for CvxNet with 50. We take this as a constraint: over-allocating cheap parts fragments structure. This is why each of our leaves gets a learned decoder rather than a fixed primitive shape. Our leaves are also not limited to sphere-like (genus-zero) topology.

PDiscoFormer relaxes the shape priors on discovered parts but explicitly leaves the choice of $K$ open.

On the volumetric side, three papers define the setting. VoxSegNet and Attention-based Part Assembly for 3D Volumetric Shape Modeling read voxel grids natively but use a fixed, supervised part list per category. Learning Hyperbolic Representations for Unsupervised 3D Segmentation is the closest prior system. It learns multi-scale volumetric embeddings without labels, but its final step is Fréchet k-means with a hand-set $k$. Our proposal targets that final step directly.

3D U-Net motivates native 3D processing over slices: 0.863 versus 0.796 IoU for a 2D U-Net on confocal volumes. SegVol supplies a pretrained 3D encoder and a strong prompted reference. However, it cannot enumerate structures on its own.

Research Program

The system has three parts.

The first is a 3D encoder that produces dense voxel features: a 3D U-Net-style CNN for $48^3$ shape grids, and SegVol's pretrained 3D ViT backbone for CT and MRI.

The second is a split head. Given a soft region mask and the pooled features of that region, it outputs a soft bipartition of the region, produced with Gumbel-Softmax so it can be trained by gradient descent, along with the two children's part codes.

The third is a shared part decoder. It reconstructs the voxels of each region from that region's code, restricted to the region's mask.

Training expands the tree to a maximum depth during the forward pass. Gradients flow only through the reconstruction terms. The stop/split decision is supervised by its own target: whether the computed gain exceeds $\lambda$. At test time, the network expands a node only while its predicted gain exceeds $\lambda$, so depth is unbounded in principle and limited in practice by the voxel count.

We will study three questions. First, does $M$ correlate with ground-truth part count across instances at fixed $\lambda$? Second, how sensitive are results to $\lambda$, compared with the sensitivity of fixed methods to $K$? Third, do the internal tree nodes correspond to real groupings, such as tumor core inside whole tumor?

Baselines

The naive reference is fixed-$K$ k-means on the same encoder's voxel features, with $K$ set to the dataset's mean part count. This is what practitioners do today. It shows how much instance-adaptivity is worth when nothing else changes.

The strongest competitor on this specific question is adaptive post-hoc clustering of the same features. We will run DP-means, which uses the same $\lambda$-per-cluster cost but applies it in feature space, and a threshold-adaptive affinity clustering in the spirit of Segment Any Primitive. Both use identical frozen features. Each method's single hyperparameter ($\lambda$ or threshold) gets the same validation search budget as our $\lambda$. Beating this baseline means a statistically higher Adjusted Rand Index and lower per-instance count error on held-out instances, averaged over three seeds.

Third, a fixed-buffer existence head in the style of DoSs: 64 slots with existence scores and Hungarian-matched reconstruction, on the same encoder and with the same training compute. This is the standard alternative for variable cardinality.

Fourth, the hyperbolic unsupervised method, using its published protocol. It reports a Dice of 0.684 on BraTS 2019, against 0.495 for the next unsupervised method (Learning Hyperbolic Representations for Unsupervised 3D Segmentation). This is the number to beat on that benchmark. On D-FAUST, the nonparametric Bayesian method reports 0.7709 IoU, and on ShapeNet it reports a Chamfer $L_1$ of 0.021. These are the reconstruction-fidelity bars for adaptive-$K$ abstraction.

As upper references, not competitors, we include SegVol run in grid-prompt "segment everything" mode (it reports 0.8593 average Dice on AMOS22 with prompts) and supervised VoxSegNet on ShapeNet Part. These tell us how far unsupervised discovery remains from supervision.

The ablations isolate each ingredient. Holding $M$ fixed tests whether the per-part price matters. Replacing the 3D encoder with a 2D per-slice encoder of matched parameters, and with a point-coordinate encoder, tests whether volumetric context matters. Scoring splits by feature distance rather than reconstruction gain tests whether the criterion matters.

Why not just…?

Why not just over-allocate slots and prune by existence score? This works for generation in DoSs. For discovery, however, the evidence from Neural Parts is that a large budget of cheap parts fragments coherent structures. A buffer also needs an imposed ordering and still caps the part count. The buffer baseline above runs this exact head on our encoder. If it matches our count accuracy, the recursion adds nothing, and we will report that outcome.

Why not just cluster good volumetric features with DP-means? This is the real threat, since it shares our cost structure. The difference is the criterion. Feature-space clustering rewards regions whose embeddings are tight. Our criterion rewards regions that a single code can reconstruct. These diverge when a geometrically coherent part has heterogeneous texture, or when two similar-looking organs are spatially separate. If DP-means on frozen features matches us, then the contribution reduces to "use good 3D features plus DP-means." In that case we would refocus the work on that simpler pipeline and its $\lambda$-transfer behavior, rather than defend the split head.

Why not just render views and use 2D foundation models? Rendering sees only surfaces, which makes it useless for tumor sub-regions or organ interiors. Even on surfaces it depends heavily on rendered normals: Segment Any Primitive loses 40.8% IoU without them. We compare against it only on ShapeNet, where it is applicable.

Why not just prompt SegVol everywhere? Grid prompting produces a variable count, but it returns overlapping masks from a supervised class vocabulary. It gives no hierarchy and no account of structures outside its 200 trained categories. We report it as a supervised ceiling and measure the gap.

Proposed Experiments

We will use four benchmarks.

ShapeNet Part voxelized at $48^3$ following the VoxSegNet protocol. Chairs with and without arms or wheels give genuine per-instance variation in part count.

D-FAUST, for reconstruction fidelity against the nonparametric Bayesian and Neural Parts numbers.

BraTS 2019 and the synthetic nested tomograms from the hyperbolic paper. Tumor sub-regions are present or absent per scan, and the synthetic set has known multi-level hierarchies.

AMOS22 or TotalSegmentator CT. The field of view differs between scans, so the number of visible organs varies naturally.

The metrics are: Adjusted Rand Index; Hungarian-matched Dice or IoU; absolute error between discovered and true part count per instance; Spearman correlation between the two counts across instances; and Chamfer distance for shapes.

The decisive check is whether, with $\lambda$ tuned once per dataset, our per-instance count correlation clearly exceeds that of fixed-$K$ k-means and the existence buffer, and at least matches DP-means on identical features, while reaching or exceeding 0.684 Dice on BraTS. A second check is cross-category transfer of $\lambda$: tune it on chairs and test on airplanes and lamps. A fixed $K$ cannot transfer this way, so this tests whether a price generalizes better than a count. The 2D-slice ablation will show whether native volumetric context, rather than the objective alone, drives the gains inside solid volumes.