When a hospital consortium pools a few hundred PET/CT or multi-sequence MRI scans and asks an unsupervised model to find patient subgroups, the most obvious structure in the data is usually not disease. It is which hospital and scanner produced the image, and which scans happen to be missing. Missingness itself tends to follow the site, because one center routinely orders PET and another rarely does. A clustering method that also trains the feature extractor makes this worse. It sharpens whatever separation is strongest, so it can confidently "discover" the hospitals. We propose a deep clustering model with two constraints. First, the cluster definitions are shared across all sites. Each cluster must look the same in every hospital, while each hospital is free to have a different share of patients in each cluster. Second, every patient must land in the same cluster whether or not a given modality is present. Clustering gradients reach only a small adapter on a frozen image encoder, which keeps the model from memorizing a cohort of 50 to 500 patients.
The distinction from simply removing site effects matters. Standardizing features within each site, which is the usual harmonization step, also erases real differences in case mix. If one center sees far more node-positive head-and-neck cancers, per-site centering subtracts part of the disease signal. Our model keeps that difference by giving each site its own mixture weights while forcing the mixture components to be site-invariant. The central falsifiable claim follows. When disease prevalence is balanced across sites, simple harmonization should match our method. When prevalence differs across sites, our method should recover clinically meaningful clusters that harmonization destroys and that unconstrained joint clustering confuses with site.
What the literature already covers
Three bodies of work sit near this question, and none of them asks it.
The first is deep clustering, which learns features and clusters together. Deep Embedded Clustering pulls embeddings toward sharpened soft cluster assignments. VaDE places a Gaussian mixture prior inside a variational autoencoder. Miklautz et al. and Chen et al. describe the feedback loop in which cluster gradients pull samples toward centroids. These papers supply our mechanism. They also motivate the problem, because the same pull that removes nuisance variation will amplify a nuisance such as scanner identity whenever that nuisance is the dominant axis of the data.
The second is incomplete multi-view clustering, which groups samples when some views (here, imaging modalities) are absent. COMPLETER recovers missing views through cross-view contrastive prediction. AIMC-CVR replaces convex anchor imputation with affine reconstruction. ACOVA shows that product-of-experts fusion double-counts correlated views. CIMLN treats unequal missing rates as a confounder and enforces consistency under counterfactual view masks. CIMLN is the closest prior work, since it targets exactly the spurious reliance on frequently observed views that site-correlated missingness creates. However, all four methods were evaluated on WebKB, MNIST, Scene-15 and similar data, with no notion of a site. CIMLN's meta-generation module also needs a sizable pool of complete samples, which small clinical cohorts lack. We keep CIMLN's counterfactual masking idea in a stripped-down form, random modality dropout on complete patients, and add the site structure it lacks.
The third is supervised work on medical shift and missing modalities. MedMAP and PASSION handle missing MRI sequences but require voxel masks. PEMMA adds PET to a frozen CT model through low-rank adapters and reports large external-site gains. Contrastive Centroid Supervision (FCCL) shows that pulling features toward class centroids improves transfer to unseen fundus cameras. FCCL is the strongest motivating evidence that shared centroids induce cross-domain alignment, but it uses labels. Our method can be read as an unsupervised, multi-modal FCCL in which the centroids are clusters and alignment is enforced explicitly per site.
Finally, two papers constrain the design. FEDD and Alhuzali et al. find that frozen backbones with light heads beat full fine-tuning when data are scarce. That rules out training a 3D encoder end to end on a few hundred patients.
Core Claim
Let $z_i$ be the fused embedding of patient $i$, $s_i$ their site, and $\mu_k, \Sigma_k$ the mean and covariance of cluster $k$, shared by all sites. Let $\pi_{s,k}$ be the share of cluster $k$ at site $s$, learned separately for each site. The soft assignment of patient $i$ to cluster $k$ is then
$$q_{ik} \propto \pi_{s_i,k}\,\mathcal{N}(z_i;\mu_k,\Sigma_k).$$
The model is trained with
$$\mathcal{L} = \mathcal{L}_{\text{drop}} + \gamma\,\mathcal{L}_{\text{clu}} + \lambda \sum_k \sum_{s\neq s'} \mathrm{MMD}\big(z \mid k,s;\; z \mid k,s'\big) - \beta\, H(\bar q).$$
The four terms do the following:
- $\mathcal{L}_{\text{drop}}$ asks a complete patient's assignment to stay the same when a randomly chosen modality is hidden.
- $\mathcal{L}_{\text{clu}}$ is a sharpened-target clustering loss in the style of Deep Embedded Clustering.
- The MMD term penalizes differences between sites in the distribution of embeddings within the same cluster, with each patient weighted by $q_{ik}$. MMD (maximum mean discrepancy) is a kernel distance between two sets of samples.
- $H(\bar q)$ is the entropy of the average assignment. It discourages all patients from collapsing into one cluster.
The key structural choice is where site-dependence is allowed. It enters only through the mixture weights $\pi_s$. A site therefore cannot claim its own cluster without paying the alignment penalty, yet it can legitimately contain more of some cluster than another site does.
Research Program
The encoders are frozen 3D encoders, one per modality, for instance the CT-pretrained UNETR backbone that PEMMA adapts. Each encoder gets a low-rank adapter and a small projection head. Available modality embeddings are averaged after projection, with no imputation in image space.
We choose averaging over product-of-experts deliberately, given ACOVA's double-counting result. We will test that choice by swapping in ACOVA's correlated fusion.
Training begins with $\gamma=0$, using only the dropout-consistency objective. Clusters are then initialized with k-means on site-centered features, and the clustering and alignment terms are switched on. The rank of the adapters is the main capacity knob. We will report results across ranks and cohort sizes rather than at one tuned point.
Proposed Experiments
Primary test bed: HECKTOR 2022. The HECKTOR 2022 benchmark has 524 labeled PET/CT patients from multiple centers in Canada, Switzerland, France and the United States. We will make the PET missing at center-dependent rates, ranging from near 0% to 80%. The clustering targets are labels derivable from the released masks without supervision leaking into training: nodal involvement (whether a GTVn label is present) and tertiles of primary tumor volume.
We will build two cohort variants. In the balanced variant, nodal prevalence is equalized across centers by subsampling. In the confounded variant, prevalence differs sharply by center.
Each variant is run at 50, 100, 200 and all patients. The metrics are:
- NMI and ARI with the target labels.
- Conditional mutual information $I(C;S\mid Y)$ between cluster and center given the target, which measures leftover site information. A raw NMI between clusters and centers would wrongly penalize genuine prevalence differences.
- Agreement of a complete patient's assignment when PET is removed at test time.
Scanner-shift test bed: AMOS. AMOS contains 600 scans from 8 scanner models and reports a cross-scanner MRI drop of over 20%. We will cluster organ-level crops, with organ identity as the target and scanner model as the site. This is a single-view stress test with unambiguous ground truth.
Sanity check: WebKB. We will run the method on WebKB with a single site, where it should reduce to an ordinary incomplete multi-view clusterer. The purpose is to confirm the site machinery does not cost accuracy in the setting prior work was built for.
Baselines
Naive reference. Extract features with the same frozen encoders, impute a missing PET embedding by the cohort mean, and run k-means. This uses exactly our features and cluster count, with no training. Any gain we report must be measured against this point.
Harmonize-then-cluster. This is the most dangerous baseline. It standardizes the frozen features within each site (subtract the site mean, divide by the site standard deviation) before k-means, using k-nearest-neighbor imputation from same-site complete patients. It receives the same features and an equal hyperparameter search over the number of neighbors.
Strongest published incomplete multi-view methods. These are CIMLN, which targets missingness bias directly, and AIMC-CVR, the strongest anchor method. Both are run on our frozen-encoder features with the authors' code and an equal tuning budget of 30 configurations per method and seed, chosen by a label-free criterion: assignment stability across seeds. COMPLETER is included because HECKTOR is two-view, the setting it was designed for.
The only absolute reference number in the file for these methods is AIMC-CVR's 91.37% clustering accuracy on WebKB, from AIMC-CVR. Matching it within noise on WebKB counts as "no loss." CIMLN reports gains of up to 4.62 points over the runner-up, but the file gives no absolute value. No reported result was found on Lacuna for unsupervised clustering on HECKTOR or AMOS, so all medical comparisons will be produced fresh under identical conditions.
What counts as beating them. On confounded HECKTOR, we need higher target NMI than the best baseline and lower $I(C;S\mid Y)$, both beyond the spread over five seeds.
Ablations. Three ablations isolate the key ingredient:
- Site-specific weights versus a single shared $\pi$. This tests whether letting prevalence vary is what protects disease signal.
- Removing the MMD alignment term.
- Removing dropout consistency.
A further ablation compares frozen encoders, low-rank adapters and full fine-tuning at each cohort size.
Why not just…?
Why not just harmonize features per site and run k-means? This is a real threat. In the balanced HECKTOR variant we expect it to be about as good as our method, and we will say so if it is. The argument against it is specific: per-site standardization subtracts each site's mean, which includes the site's disease mix. It should therefore lose nodal-status NMI exactly in the confounded variant. If it does not, our core claim fails. In that case the right conclusion is that harmonization suffices, and the contribution shrinks to the missingness-consistency part.
Why not just run an existing incomplete multi-view clusterer such as CIMLN? CIMLN addresses spurious reliance on frequently observed views, which is half of our problem. But it has no notion of site, so nothing stops it from forming site-shaped clusters. Its meta-generation step also assumes many complete samples. The 50-to-200 patient runs, with site-correlated PET missingness, test both weaknesses directly through $I(C;S\mid Y)$ and target NMI.
Why not just discard patients missing PET? Under site-correlated missingness, dropping incomplete cases removes some centers almost entirely, which shrinks the cohort and changes its case mix. PASSION documents how training only on complete cases fails when a modality is absent at test time. We include a complete-case k-means run to quantify this on HECKTOR.
Why not fine-tune the whole encoder with the clustering loss? FEDD and Alhuzali et al. both find full fine-tuning inferior to frozen backbones with light heads at small sample sizes. With an unsupervised objective, overfitting means memorizing scanner texture. The adapter-rank-versus-cohort-size ablation will show where, if anywhere, full fine-tuning catches up.
Decisive Checks
The proposal stands or falls on the balanced-versus-confounded contrast on HECKTOR. Our method should tie harmonize-then-cluster when prevalence is balanced and beat it on target NMI when prevalence is confounded. It should also beat CIMLN and AIMC-CVR on $I(C;S\mid Y)$ in both variants.
A second check is that removing site-specific mixture weights should reproduce harmonization's failure in the confounded variant. That would show the gain comes from letting prevalence differ, not from extra capacity.
A third check is that the AMOS organ-crop clusters should not group by scanner model. Measured as $I(C;S\mid Y)$, they should carry less scanner information than the naive frozen-feature clustering, at no loss of organ NMI.
References
- Unsupervised Deep Embedding for Clustering Analysis
- Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
- Deep Embedded Non-Redundant Clustering
- Relaxed multi-view clustering in latent embedding space
- COMPLETER: Incomplete Multi-View Clustering via Contrastive Prediction
- Alleviate Anchor-Shift: Explore Blind Spots with Cross-View Reconstruction for Incomplete Multi-View Clustering
- Beyond Independence: Learning Correlated Views for Variational Incomplete Multi-View Clustering
- Alleviating Observation Bias via Causal-Invariant Meta-Learning for Unbalanced Incomplete Multi-view Clustering
- MedMAP: Promoting Incomplete Multi-Modal Brain Tumor Segmentation With Alignment
- PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates
- PEMMA: Parameter-Efficient Multi-Modal Adaptation for Medical Image Segmentation
- Contrastive Centroid Supervision Alleviates Domain Shift in Medical Image Classification
- FEDD - Fair, Efficient, and Diverse Diffusion-Based Lesion Segmentation and Malignancy Classification
- Predicting Sign of Depression via Using Frozen Pre-trained Models and Random Forest Classifier
- Automated Head and Neck Tumor Segmentation from 3D PET/CT HECKTOR 2022 Challenge Report
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation