Vision models pretrained on huge collections of photographs see the world one flat image at a time. Much of the data we most want to understand has a third axis instead. A CT scan is a stack of slices through a body, and an endoscopic surgery video is a stack of frames through time. Today the best way to carry a 2D model over to such data is to insert small trainable 3D modules, called adapters, that let each slice or frame look at its neighbours. These adapters are trained with dense manual masks, and those masks are exactly what volumetric and surgical data lack. We propose to train the same adapters with no labels at all. The adapter's only job becomes grouping every image patch in the stack into a fixed number of clusters, and it is rewarded when those groupings stay consistent along the third axis.
The bet is simple. A single slice or frame is often ambiguous: a low-contrast organ boundary, or a kidney hidden under fat. The neighbouring slices usually are not. If the clustering signal can only be satisfied by looking across the third axis, the adapter should learn volumetric context that post-hoc smoothing cannot supply. One mechanism covers both depth in CT/MRI and time in surgical video. That lets us test whether "image-to-volume" and "image-to-video" are really the same adaptation problem.
Core Claim
Joint representation-and-clustering training can supervise parameter-efficient 3D adapters placed on a frozen 2D pretrained encoder. "Joint" here means the clustering loss backpropagates into the feature extractor rather than being run once on fixed features. Under this training the adapters produce unsupervised part/organ segmentations of volumes and videos that beat per-slice clustering of the same frozen features. They also beat post-hoc 3D smoothing of those clusterings. The gains should concentrate on structures that are ambiguous within a single slice.
Two design choices make this more than "add a clustering loss." First, the 2D backbone stays frozen and only the 3D adapter and cluster prototypes train. The pretrained image features therefore anchor the solution and limit the self-reinforcing errors that plague self-trained clustering. Second, the balanced-cluster assumption in optimal-transport clustering is replaced by learned cluster proportions, because a liver and an adrenal gland differ in volume by orders of magnitude.
Relation to Prior Work
Three papers supply the method. DEC introduced the idea of backpropagating a clustering loss into an encoder. The entropic optimal-transport formulation of Genevay et al. avoids collapse into one cluster by solving a Sinkhorn problem with fixed cluster-size marginals, which lets training start without an autoencoder warm start. Temporal Optimal Transport extends this online clustering to video frames, with a coherence loss that pulls adjacent frames together. We reuse the Sinkhorn machinery and the idea of neighbour coherence. We differ in three ways. We cluster dense patch tokens rather than whole frames. We drop the equal-size and monotonic-order assumptions that paper itself flags as failure modes. And our coherence acts through a 3D adapter rather than on 2D frame embeddings.
Two papers motivate the proposal by showing the gap from the other side. MA-SAM inserts 3D convolutional adapters into a frozen 2D Segment Anything encoder, reshaping tokens into a 3D tensor, convolving across depth or time, and reshaping back. It works for both CT/MRI and surgical video, but only with dense masks or prompts. We keep its adapter architecture and replace its supervision with clustering. ZeroI2V similarly repurposes attention heads to look across frames, again fully supervised.
The closest unsupervised work connects clustering and video but not volumetric features. ODCT stabilizes online clustering by averaging memory features along tracks from an external tracker. It reaches 32.95% ImageNet linear top-1 versus 23.14% for plain online clustering. That is evidence that third-axis consistency helps clustering. However, its consistency is imposed on 2D crops through a non-differentiable tracker. In our version the consistency is learned inside the network.
Two papers constrain the design. CycDA argues that forcing images and videos into one feature space jointly causes conflicts, and so separates adaptation into stages. Our frozen-backbone-plus-adapter split is a partial concession to that finding: the 2D appearance space is never overwritten. Li et al. show that naively inflating 2D weights into 3D entangles appearance with motion. Keeping appearance in the frozen 2D path and depth/time in a separate adapter follows their separable-channel logic.
The volumetric literature explains why labels are the bottleneck. SegVol documents how fragmented and partial public 3D annotations are. AMOS reports drops of more than 20% on unseen scanners. Both point to the value of adapting on unlabeled target scans.
Research Program
The model is a frozen 2D pretrained ViT encoder $f$ applied slice-wise or frame-wise, with MA-SAM-style 3D adapters $A$ inserted between blocks. A set of $K$ learnable prototypes $c_1,\dots,c_K$ sits on the output tokens. For a sub-volume (a stack of $D$ slices or frames), let $z_n$ be the adapted feature of patch token $n$. Predicted assignments are $p_{nk} \propto \exp(z_n^\top c_k/\tau)$, where $\tau$ is a temperature. Target assignments $q$ come from Sinkhorn on a second augmented view of the same stack. In that Sinkhorn step, the cluster-size marginal is a learned vector $b$ on the simplex, penalized toward uniform only by a KL term with weight $\lambda$, rather than being fixed to uniform. The training loss is
$$ \mathcal{L} = -\sum_{n}\sum_{k} q^{(2)}_{nk}\log p^{(1)}_{nk} \;-\; \sum_{n}\sum_{m \in \mathcal{N}_z(n)}\sum_k q_{mk}\log p^{\setminus z}_{nk}, $$
where superscripts denote the two views and $\mathcal{N}_z(n)$ is the same spatial location in the adjacent slices or frames. $p^{\setminus z}_{nk}$ is the assignment predicted for token $n$ when its own slice is masked at the adapter input, so that $n$'s features come only from the frozen 2D path of its neighbours, mixed by $A$.
The second term is the key ingredient. It asks the adapter to predict a token's cluster from its neighbours along the third axis. That task is only solvable if the adapter carries real depth or time context. For video, where camera motion breaks the same-location correspondence, $\mathcal{N}_z$ is widened to a small spatial window.
The program has three stages. First, we build and train this on unlabeled CT (BTCV and AMOS22 CT) and unlabeled surgical video (EndoVis18), using labels only for evaluation. Second, we analyze what the adapter learned. We measure per-structure gains against single-slice ambiguity, quantified as the Dice of per-slice clustering for that structure. We also measure how cluster assignments flip across adjacent slices. Third, we test the image-to-volume versus image-to-video question. We train one adapter on CT, one on video, and one on both, and cross-evaluate. A shared third-axis mechanism predicts positive transfer. Separate mechanisms predict none.
Baselines
The naive reference is the frozen 2D encoder run on each slice independently, followed by $k$-means on patch tokens with the same $K$. This measures how much structure the image-pretrained features already contain, with no adaptation and no depth context.
The strongest label-free competitor is that same per-slice clustering followed by 3D post-hoc regularization. We apply a 3D conditional random field, or majority smoothing across a window of $D$ slices, tuned on the same validation split with the same number of trials as our method. This competitor is the right one because it supplies volumetric context without touching the representation. If it matches us, then third-axis context need not enter the features at all.
The second strong competitor is two-stage adapter training. The identical 3D adapter is trained with a self-supervised objective that has no clustering term, then $k$-means is run on its outputs. The objective combines masked reconstruction and contrastive learning across neighbouring slices or frames, in the style of ViC-MAE. This isolates joint clustering from 3D context: the architecture, data and compute are the same, and only the objective differs.
The third competitor is a 2D-only coherence baseline in the spirit of TOT+TCL and ODCT. A lightweight 2D adapter is trained with our Sinkhorn loss plus a feature-level pull between adjacent slices, but with no 3D convolution.
Within our method, ablations remove, one at a time, the masked-neighbour prediction term, the learned marginals (reverting to uniform), and the frozen backbone (full fine-tuning). All runs share one GPU budget, crop size, $K$, and hyperparameter search size.
For supervised ceilings there is no reported unsupervised segmentation result found on Lacuna for BTCV, AMOS22 or EndoVis18. We will contextualize with supervised numbers from the file. These are a 3D U-Net at 89.04% mDice on AMOS CT (Ji et al.), SegVol at 0.8593 average Dice on AMOS22 (SegVol), and the EndoVis18 covered kidney at mIoU 0.224 (Allan et al.).
Beating the field means two things. First, a statistically reliable gain in Hungarian-matched mean Dice over both the post-hoc-smoothing and the two-stage-adapter baselines, across three seeds. Second, gains that are larger on structures with low single-slice Dice than on easy ones.
Why not just…?
Why not just cluster frozen features per slice and smooth in 3D afterwards? This is the real threat, especially on CT, where adjacent slices are nearly aligned and smoothing is cheap. The file offers indirect evidence against it. Allan et al. show frame-wise models collapsing on occluded tissue, and Dou et al. show that adjacent-slice context inside 3D kernels resolves low-contrast boundaries. Neither result is unsupervised, though. Smoothing can only fix a slice whose neighbours are already correct; it cannot create a boundary absent from the per-slice features. We therefore predict our advantage appears on ambiguous structures and vanishes on easy ones. If smoothing matches us there too, we will report that volumetric context does not need to enter the representation for label-free segmentation.
Why not just pretrain the adapter self-supervised and cluster offline? DEC reports 84.3% versus 53.5% for autoencoder-plus-$k$-means on MNIST. TOT+TCL beats the two-stage CTE by 11.9% F1 on 50 Salads. These results suggest decoupled objectives waste capacity on reconstruction detail. Neither result concerns dense 3D medical tokens, however, so the matched two-stage-adapter baseline tests this directly.
Why not just add a temporal coherence loss to a 2D model and skip 3D adapters? Pulling adjacent features together rewards sameness, not prediction. It can be satisfied by blurring features along the axis, which hurts thin structures. The 2D-coherence baseline, and the ablation removing masked-neighbour prediction, show whether prediction through a 3D adapter adds anything beyond coherence.
Why not just use prompts with a supervised adapter like MA-SAM? When a few labels or clicks exist, that is the better tool. MA-SAM's prompted pancreas-tumor Dice rises from 41.6% to 80.3%. Our regime is unlabeled target data, such as new scanners or new surgical procedures, where the adapter must be adapted before anyone annotates. We will also test whether our label-free adapter is a better initialization for MA-SAM fine-tuning with 1–5 labeled volumes.
Proposed Experiments and Decisive Checks
We train on unlabeled BTCV and AMOS22 CT, and on EndoVis18 video frames. We evaluate unsupervised segmentation by Hungarian matching of $K$ clusters to ground-truth classes, reporting mean Dice for CT and mIoU for EndoVis18. We also report an over-clustered variant with many-to-one matching, and adjacent-slice assignment flip rate.
The first decisive check is the stratified analysis. We rank structures by per-slice-clustering Dice and plot our gain over post-hoc smoothing against that rank. A negative slope supports the claim; a flat zero line refutes it. Candidate hard cases include the pancreas and adrenals in CT, and the covered kidney in EndoVis18.
The second decisive check is scanner shift. We adapt on unlabeled AMOS MRI from vendors held out of any labeled source, then compare against the frozen baseline. AMOS reports drops of more than 20% in this setting, which gives room for a label-free fix.
The third decisive check is cross-axis transfer. We compare the CT-trained adapter applied to video, the video-trained adapter applied to CT, and joint training. This tells us whether one adapter can learn "context along the third axis" for both depth and time, or whether the two only look alike on paper.
References
- Unsupervised Deep Embedding for Clustering Analysis
- Differentiable Deep Clustering with Cluster Size Constraints.
- Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering
- MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation
- ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video
- Online Deep Clustering with Video Track Consistency
- CycDA: Unsupervised Cycle Domain Adaptation to Learn from Image to Video
- In Defense of Image Pre-Training for Spatiotemporal Recognition
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- ViC-MAE: Self-supervised Representation Learning from Images and Video with Contrastive Masked Autoencoders
- 2018 Robotic Scene Segmentation Challenge
- 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes