Supervoxel Tokens That Split Where the Boundary Is Unclear

Segmenting soft-tissue structures in 3D CT, such as the pancreas or a coronary artery on a scan without contrast dye, forces two needs against each other. The model needs global context, meaning knowledge of where the stomach, spine and heart sit across the whole volume, because the local intensity edge between organ and neighbour is faint or missing. It also needs full voxel resolution at exactly those faint edges. Current systems give up one or the other. Some cut the volume into 2D slices and lose continuity across slices. Some crop small 3D patches and lose the global view. Some downsample the whole grid and smear the thin structure into its surroundings. We propose to represent the volume as a few thousand supervoxels, compact irregular blobs of similar-looking voxels, so that a global model can see the entire scan at once. The network then breaks any supervoxel it is unsure about back into individual voxels, so that ambiguous boundaries are always handled at full resolution.

The idea answers a known failure of region-based reduction. If a blob straddles an object boundary, nothing downstream can split it, and on low-contrast tissue intensity-based clustering creates exactly such straddling blobs. We make the grouping learned, supervised to respect the true labels rather than intensity edges, and reversible where the model is uncertain. We borrow the "mixed region gets refined" logic of classical coarse-to-fine energy minimisation and move it inside a deep 3D network. The claim is testable. At a matched number of tokens, adaptive, splittable supervoxel tokens should beat rigid grid tokens and fixed intensity supervoxels on boundary accuracy for small low-contrast targets. They should also remove most of the accuracy ceiling that fixed supervoxels impose.

Where the field stands and what is missing

Region-based reduction is well established as a way to make global reasoning tractable. Over-segmenting an image into regions cuts the number of graphical-model variables by orders of magnitude (Learning and Incorporating Top-Down Cues in Image Segmentation). Adaptive regions stop at natural edges where rigid downsampling cuts across them (Superpixels, Occlusion and Stereo). Superpixel graphs make exact Laplacian computation feasible in hyperspectral imaging (Superpixel Contracted Graph-Based Learning for Hyperspectral Image Classification). In video they give about a tenfold reduction in compute (GraphVid). These works motivate the proposal. The same literature also constrains it. The top-down-cues line notes that bottom-up regions set an error floor no downstream model can undo. Intensity-based clustering such as SLIC either bleeds across or shatters weak borders, and weak borders are exactly the regime of pancreas and non-contrast arteries.

Four prior works sit closest to our design. Pieces-of-parts for supervoxel segmentation with global context groups DCE-MRI voxels into perfusion supervoxels and corrects local errors with a parts-based graphical model, reaching a median DSC of 0.63. Its supervoxels are static, hand-built and classified by linear discriminant analysis, so the boundary errors they contain are permanent. MSP averages deep features within multi-scale SLIC superpixels to snap 2D predictions to edges, gaining 0.90 mIoU on ADE20K. Its flow is one-way, since fixed SLIC boundaries guide features but are never corrected, and it is 2D natural-image only. Coarse-to-Fine Semantic Video Segmentation Using Supervoxel Trees supplies the mechanism we reuse. It labels coarse regions as pure or "mixed", and refines only the mixed ones at finer scale, for a 3–10x speedup on video. It does this with handcrafted energies on RGB video, not learned features on CT. Efficient Connectivity-Preserving Instance Segmentation with Supervoxel-Based Loss Function uses supervoxels only inside a training loss for topology, not as the units the network computes on.

On the context side, prior systems that handle low contrast either drop the third dimension or pay for dense voxel computation. DB-MSMUNet combines Mamba state-space blocks with deformable convolutions and a separate edge-decoding branch, and reports 89.47% DSC on NIH Pancreas-CT and 87.59% on MSD Pancreas. A state-space block, such as Mamba, is a sequence layer that carries context linearly in sequence length. DB-MSMUNet works on 2D slices and states that it does not explore 3D consistency. PanSAM adds a 3D convolutional aggregator after a frozen 2D Segment Anything encoder and reaches 87.01% DSC on NIH Pancreas-CT. Its encoder never sees out-of-plane context. NA-UNETR stays fully 3D with local and dilated neighbourhood attention for the non-contrast LAD artery, reporting 45.64% DSC and 38.16 mm HD95 on LAD-SEG versus 42.54% for nnU-Net. It pays for dense voxel attention and needs pretraining on contrast CT. HybridTM shows, on point clouds, that flattening 3D data into a 1D sequence for Mamba destroys fine geometry unless local attention is added. This is a constraint that pushes us to compute global context on a graph rather than a serialized scan.

Core claim

The irreversible-boundary problem of supervoxel reduction comes from two removable choices: grouping by raw intensity, and treating the grouping as final. If supervoxels are formed from learned features and trained to be label-pure, and if every supervoxel the model judges impure is dissolved back into voxels before the final prediction, then supervoxel tokens can give whole-volume global context at a token count near rigid downsampling. They do so without the boundary damage of rigid downsampling or the error ceiling of fixed supervoxels. The gains should concentrate where contrast is lowest and structures are smallest, and should show up most in boundary metrics and slice-to-slice continuity rather than in bulk overlap.

Research program

The network has four stages, and the last three repeat a small number of times. First, a shallow full-resolution 3D convolutional stem produces per-voxel features. It uses in-plane-first kernels to respect anisotropic spacing such as 0.7 mm in-plane against 2.5–5 mm slices. Second, a differentiable soft clustering step, a few iterations of SLIC-style assignment in learned-feature-plus-physical-millimetre space, assigns each voxel softly to one of $K$ supervoxels. $K$ is a few thousand to about sixteen thousand per scan. Because distances are in millimetres, supervoxels naturally span fewer slices when slices are thick. Third, a global module computes context over the supervoxel adjacency graph, with neighbours defined by shared voxel faces. It uses attention restricted by graph distance plus a small set of global tokens, which avoids the 1D serialization that HybridTM shows to be harmful. Fourth, each supervoxel receives a coarse class distribution and a "mixed" score. Supervoxels with the highest mixed scores, up to a fixed voxel budget, are dissolved. Their voxels are refined by local 3D neighbourhood attention in the style of NA-UNETR, conditioned on the global features broadcast from the graph. Confident supervoxels pass their label to their voxels through the soft assignment.

On later passes the clustering is recomputed from the refined features. Grouping therefore improves as semantics improve, which is the feedback that MSP and Pieces-of-parts lack. This echoes the iterative refinement of RecycleNet, with the difference that here the token structure changes between passes, not just the features.

The ingredient that makes grouping cut along borders where intensity does not is a purity loss. Let $Q_{vk}$ be the soft assignment of voxel $v$ to supervoxel $k$, and let $y_v$ be the ground-truth label of voxel $v$. The label mixture inside supervoxel $k$ is $p_k(c)=\sum_v Q_{vk}\,[y_v=c] / \sum_v Q_{vk}$, the assignment-weighted fraction of its voxels with label $c$. We penalise

$$\mathcal{L}_{\text{pure}} = \sum_k \frac{\sum_v Q_{vk}}{\sum_{v,k'} Q_{vk'}}\; H(p_k),$$

which is the size-weighted entropy $H$ of each supervoxel's label mixture. The mixed score used at test time is the predicted version of this quantity, the entropy of the supervoxel's predicted class distribution. It is trained to match $H(p_k)$. The total loss is Dice plus cross-entropy on final voxel predictions, plus $\mathcal{L}_{\text{pure}}$, plus the mixed-score regression.

A diagnostic runs alongside the model: achievable segmentation accuracy. This is the Dice obtained by giving every supervoxel the majority ground-truth label. It measures the error ceiling of any partition directly. It lets us show, before any model comparison, whether intensity SLIC on pancreas and LAD scans really imposes the ceiling the literature describes, and how much the learned, purity-trained, splittable partition removes.

Baselines

The naive reference is a 3D nnU-Net at full resolution with its standard patch-based training and its own configuration rules. The AMOS benchmark shows that a plain 3D U-Net reaches 89.04% mDice on CT, ahead of Swin-UNETR at 86.32%, so a plain 3D convolutional baseline is a real competitor and not a straw man. We also run the nnU-Net low-resolution-to-full-resolution cascade. It is the most direct non-graph way to give a 3D network global context, so it must be beaten, not only the single-stage model.

On the low-contrast vessel task the strongest existing method is NA-UNETR, with 45.64% DSC and 38.16 mm HD95 on LAD-SEG, versus nnU-Net at 42.54% as reported in that paper. We use the same ImageCAS pretraining and low-rank fine-tuning protocol for our model, so that the pretraining advantage is shared. Beating it means higher DSC and lower HD95 on the same cross-validation folds, at no more than its compute.

On pancreas the strongest reported numbers are DB-MSMUNet's 89.47% DSC on NIH Pancreas-CT and 87.59% on MSD, and PanSAM's 87.01% DSC on NIH Pancreas-CT (DB-MSMUNet; PanSAM). Their splits and preprocessing differ from ours, so we retrain both from public code where available on our folds rather than compare against the printed numbers. For pancreas, success means matching the retrained 3D nnU-Net in DSC while clearly improving surface Dice at 1–2 mm tolerance, HD95, and slice-continuity measures. We do not expect a large bulk-overlap gain on pancreas and would not claim one.

Three ablations isolate the key ingredient, each at an identical token count, parameter count and training budget. The first is a rigid-token ablation that replaces supervoxels with a regular grid of the same size. It tests whether adaptivity matters at all. The second is a static ablation that uses intensity SLIC supervoxels, fixed and never split. It is in effect a deep version of Pieces-of-parts and MSP, and tests whether learned grouping and splitting matter. The third is a no-split ablation that keeps learned purity-trained grouping but dissolves nothing, and tests the splitting mechanism specifically. All models get the same GPU memory cap, the same number of training epochs and the same hyperparameter search budget of a fixed number of trials.

Why not just…?

Why not just use a 3D nnU-Net, possibly in its cascade form? This is the real threat, and on large, reasonably contrasted organs it may simply tie. The case against it rests on thin, low-contrast targets. On LAD-SEG, nnU-Net trails neighbourhood attention by about three DSC points (NA-UNETR). The cascade's low-resolution stage is exactly the rigid downsampling that blurs a vessel a few voxels wide. We settle it directly: if our model does not beat the cascade on LAD-SEG HD95 and pancreas surface Dice at matched compute, the proposal fails on its central claim.

Why not just process 2D slices with a strong pretrained encoder and add a 3D aggregation head? PanSAM does this and is competitive. Its encoder still forms every feature without out-of-plane information, and it loses about two DSC points with automated rather than ground-truth prompts. Slice-wise models are known to produce fragmented 3D shapes (Deep convolutional networks for pancreas segmentation in CT imaging). We measure slice-to-slice consistency: the count of disconnected components per case and the variance of per-slice Dice across consecutive slices. If the 2D-plus-head approach matches us on these, volumetric tokenization adds nothing here.

Why not just downsample the volume to a fixed grid small enough for global attention? The rigid-token ablation answers this at equal token count. Prior work argues grid cells cut across edges (Superpixels, Occlusion and Stereo). A pancreas covering under 1.5% of a slice, or a millimetre-scale artery, becomes a partial-volume mixture at coarse resolution. If grid tokens plus the same dissolve-and-refine step do as well as supervoxels, then splitting is the active ingredient and adaptive grouping is not. We would report that and simplify the method accordingly.

Why not just snap predictions to superpixels afterward, as MSP or a dense CRF does? Post-hoc snapping inherits the partition's errors. MSP's authors note it cannot recover an edge that SLIC missed (MSP). The achievable-accuracy diagnostic quantifies this ceiling for intensity supervoxels on pancreas and LAD. The static ablation shows how much of it survives inside a trained network.

Proposed experiments

Training and evaluation use NIH Pancreas-CT and MSD Pancreas with fixed five-fold splits, LAD-SEG with ImageCAS pretraining as in NA-UNETR, and AMOS CT as a multi-organ check covering both easy organs and hard ones such as the adrenal glands and duodenum. Metrics are DSC, HD95, surface Dice at 1 mm and 2 mm, the slice-continuity measures above, peak GPU memory, and FLOPs per volume.

Three checks decide the question. The first is the ceiling check. Achievable segmentation accuracy of intensity SLIC should fall well short of 100% on pancreas and LAD at our $K$, and the learned, purity-trained, splittable partition should close most of that gap. If intensity SLIC already sits near 100%, the irreversible-bottleneck motivation does not hold in this regime. The second is the matched-budget check: supervoxel tokens should beat rigid grid tokens and fixed SLIC tokens at equal $K$, with the margin growing as contrast falls. We plot the margin per AMOS organ against measured boundary intensity contrast; the prediction is a negative slope. The third is the budget sweep: varying the dissolve budget from zero to a large fraction of voxels should show boundary error falling quickly and then flattening. That shape would show a small fraction of uncertain supervoxels carries most of the boundary difficulty, as the mixed-label pruning in Supervoxel Trees found for video.

A model that wins only on bulk Dice and not on these three checks would not support the claim.