When someone clusters an unlabeled image collection with a pretrained encoder, they face many choices: which encoder, whether to reduce dimensions first, which clustering algorithm, and how many clusters. No labels exist to settle these choices. The usual fix is an internal index such as the Silhouette score, which rewards tight, well-separated clusters measured by distances in the embedding space. That works poorly for the question people actually ask, which is "which of these pipelines should I trust?" The candidate pipelines live in different spaces with different scales and dimensions, so their distance-based scores cannot be compared fairly. The published evidence also disagrees. On foundation-model embeddings, Silhouette computed after a nonlinear reduction tracks clustering quality (An Empirical Study into Clustering of Unseen Datasets with Self-Supervised Foundation Models). On selection across diverse model families, Silhouette is statistically no better than picking at random (The Need for Unsupervised Outlier Model Selection).
We propose to score a pipeline by something that lives in label space rather than distance space. Split the data into two disjoint halves, run the whole pipeline independently on each, carry the first half's cluster labels over to the second half with a simple nearest-neighbour classifier, and measure how many bits of mutual information survive between the carried-over labels and the second half's own labels. A pipeline gets credit only for structure that is reproducible from an independent sample. Because the score is mutual information rather than a normalized agreement, it also grows with the number of distinguishable clusters. This is a tractable, deep-embedding version of the two-sample "approximation capacity" idea from information-theoretic model validation. We expect it to rank pipelines across encoders where geometric indices cannot, and we test that claim on the 26-dataset zero-shot clustering suite and on self-supervised checkpoint selection for dense prediction.
Core Claim
For a candidate pipeline $\pi$ (encoder, reduction, algorithm, hyperparameters including the number of clusters $k$), draw disjoint halves $S_1, S_2$ of the data and let $\pi(S_j)$ be the labels the pipeline assigns when run on half $j$ alone. Let $f_1$ be a $k$-nearest-neighbour classifier fit in $\pi$'s own space to $(S_1, \pi(S_1))$. The score is
$$T(\pi) = \mathbb{E}_{\text{splits}}\left[\hat I\big(f_1(S_2);\, \pi(S_2)\big) - \mathbb{E}_{\text{perm}}\,\hat I\big(f_1(S_2);\, \sigma(\pi(S_2))\big)\right].$$
Here $\hat I$ is the plug-in mutual information in bits, and $\sigma$ is a random permutation of the second half's labels. The permutation term removes the positive bias that plug-in mutual information has with many small clusters.
The score has three properties that motivate the claim. First, a partition imposed on structureless data shifts arbitrarily between halves, so $T$ falls toward zero. Second, a stable but coarse partition is capped at $\log_2 k$ bits, so $T$ does not collapse toward $k=2$ the way normalized stability scores do. Third, $T$ touches geometry only through the pipeline's own outputs, so comparing a DINO-plus-UMAP pipeline with a MAE-plus-PCA pipeline requires no shared distance scale.
The claim being tested is that $T$ gives lower selection regret than Silhouette and density-based indices when candidates span different encoders and reductions. Within a single embedding space, geometric indices may stay competitive, and we would report that if it happens.
Relation to Prior Work
Two papers supply the method. Context Sensitive Information: Model Validation by Information Theory frames validation as communication between two independent samples. It scores a model by the mutual information between solution sets under a Boltzmann temperature, which requires partition functions that are only tractable with annealing or MCMC (Markov chain Monte Carlo sampling). We keep the two-sample, information-counting idea and drop the partition function: hard assignments plus a transfer classifier replace the posterior over solutions, and a sweep over $k$ plays the role of the temperature sweep. Maximin Separation Probability Clustering argues that real structure should yield pseudo-labels a classifier can learn. It achieves invariance to linear rescaling, but only for two clusters. We share the "labels should transfer" intuition but measure transfer to an independently clustered sample rather than to held-out points of the same partition. The difference matters because a $k$-means partition is linearly separable by construction, so held-out learnability alone says little.
Three papers motivate the problem. The Lowe et al. study and its companion Zero-shot Clustering of Embeddings with Pretrained and Self-Supervised Learnt Encoders benchmark clustering across 26 unseen datasets. They recommend Silhouette in 50-dimensional UMAP space (UMAP is a nonlinear method that reduces dimensionality while preserving local neighbourhoods) as a label-free proxy. The file reports no numerical Silhouette-to-AMI correlation for that suite, and those papers do not separate within-space from cross-space ranking, which is exactly the comparison we isolate. Ma et al. motivate from the other side: stand-alone internal indices fail across model families, and random noise can earn high clustering scores.
Two papers constrain the design. The DBCV index is more informative than DCSI, CDbw, and VIASCKDE indices shows that convex-biased indices mislead on concave and density-shaped clusters. DBCV (density-based cluster validity) tracks ARI best there, but inconsistently on some real data: 33% trend consistency on Diabetes type one. We therefore treat DBCV as a required competitor rather than assuming geometry-free scores win. Exploring Structural Degradation in Dense Representations for Self-supervised Learning builds DSE (Dense Representation Structure Estimator) from $k$-means separability plus effective dimensionality and uses it to pick self-supervised checkpoints. Its fixed $k$ and convexity assumption are the limitations our score is meant to remove.
Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation is the closest active-validation method. It probes a classifier with mixup interpolations between pseudo-labelled points. It needs a trained predictor and a fixed label set, so it does not directly select clustering pipelines with unknown $k$.
Research Program
We would build the estimator, with the transfer classifier, the permutation correction and averaging over repeated splits. We would then analyse its behaviour on synthetic data: isotropic Gaussian noise, well-separated blobs, the concave half-moon and circle shapes from the DBCV study, and anisotropically rescaled copies of each. The synthetic study should confirm three predictions. $T$ should be near zero on noise, where Silhouette can be high. $T$ should peak near the true $k$ on blobs. $T$ should follow whatever the clustering algorithm recovers on concave data, high for HDBSCAN and low for $k$-means, rather than imposing its own convexity bias.
The main study then applies $T$ to pipeline selection over real foundation embeddings. A secondary study replaces DSE's separability term for dense checkpoint selection: the patch embeddings are clustered on two disjoint image subsets, so the score asks whether patch-level structure is reproducible.
Three design questions are worth answering explicitly. The first is whether the transfer classifier should be fit in the pipeline's reduced space or its raw encoder space. The second is how many splits are needed before rankings stabilize; we expect the variance of the plug-in estimate to shrink roughly as one over the number of split pairs, since splits are near-independent. The third is whether soft assignments with a temperature, closer to the original approximation-capacity formulation, add anything over hard labels.
Baselines
The naive reference point is a fixed default pipeline chosen without looking at the data: DINO ViT-B, UMAP to 50 dimensions, and Ward agglomerative clustering, which the Lowe et al. study identifies as the best overall configuration. Here $k$ is set by a fixed rule (the median $k$ over the candidate grid). We also report random selection from the pool and the oracle that picks the pipeline with the highest AMI, which bounds the achievable gain.
The strongest existing competitor on this question is Silhouette in 50-dimensional UMAP space, as recommended by An Empirical Study into Clustering of Unseen Datasets with Self-Supervised Foundation Models. It is strongest because it was validated on exactly this suite and is what practitioners would use. We compute it for every candidate on the same data, in the same UMAP space each pipeline uses, and for pipelines without UMAP both in their native space and after a shared UMAP-50 projection, so Silhouette gets its best case. No reported result was found on Lacuna for a numerical Silhouette-to-AMI correlation or selection regret on this suite, so we will reproduce it. Beating it means lower mean AMI regret to the oracle across the 26 datasets, with a paired Wilcoxon test at $p<0.05$, and specifically on the cross-encoder pool.
DBCV is the density-aware competitor (The DBCV index is more informative), computed in the same spaces. It matters because HDBSCAN is in the pool and produces non-convex clusters that Silhouette penalizes unfairly. Davies-Bouldin and Calinski-Harabasz are reported for completeness.
For dense checkpoint selection, the baseline is DSE (Exploring Structural Degradation in Dense Representations for Self-supervised Learning). That paper reports a Kendall's $\tau$ of about 0.57 with downstream performance and a 3.0% average mIoU gain from DSE-guided selection across 16 methods on COCO-Stuff, PASCAL VOC, ADE20k and Cityscapes. We run both scores on the same checkpoints, with the same roughly 0.16% data budget and equal GPU time. Beating DSE means a higher $\tau$ or a higher average selected mIoU. Matching it without a fixed $k$ would count as partial success.
Two ablations isolate the key ingredient. The first replaces mutual information with normalized agreement (AMI between halves), which removes the expressiveness reward. The second replaces the independent re-clustering of $S_2$ with held-out accuracy of $f_1$ against the full-data partition, which removes the two-sample independence. If either ablation matches the full score, our explanation of why it works is wrong.
Why not just…?
Why not just use Silhouette in UMAP space, since it already works on this suite? It may well work within a single embedding space, and we expect it to stay competitive there. The open question is cross-space comparison: Silhouette values from a ResNet-50 MAE embedding and a ViT-B DINO embedding sit on different distance scales. Ma et al. found that clustering indices selecting across heterogeneous models are indistinguishable from random. Our experiment reports Kendall's $\tau$ separately for within-encoder and across-encoder pools. If Silhouette holds up across encoders too, the main claim fails and we will say so.
Why not just use DBCV, which handles non-convex clusters? DBCV fixes the convexity bias but is still a distance-based score in a particular space, and its own evaluation shows uneven reliability on real data (33% ARI-trend consistency on two of five EHR datasets). DBCV is a real threat on the HDBSCAN-heavy part of the pool. The per-algorithm breakdown will show whether $T$'s advantage, if any, comes from the cross-space comparison rather than from cluster shape.
Why not just use classical resampling stability, such as agreement between clusterings of two subsamples? This is the most serious alternative, because our score is a variant of it. Normalized stability is known in practice to favour coarse partitions, since two clusters are easy to reproduce. The un-normalized mutual information is exactly what is supposed to counter that bias. The normalized-agreement ablation tests this directly. If it matches $T$, the contribution reduces to "use stability with UMAP pipelines," which is still useful but much narrower, and we would report it that way.
Why not just pick the default pipeline? Ma et al. show that defaults can be hard to beat: consensus methods tied a random-parameter Isolation Forest even though that forest was only about the 84th best of 297 models. The oracle-versus-default gap on our suite measures how much room selection has. If that gap is small, no selector matters.
Proposed Experiments
The main testbed is the 26-dataset zero-shot suite from Lowe et al., spanning in-domain, domain-shifted, near-OOD, fine-grained and far-OOD tiers. The candidate pool crosses supervised, MoCo-v3, DINO, VICReg and MAE encoders on ResNet-50 and ViT-B with no reduction, PCA, and UMAP-50. The algorithms are $k$-means, Ward and HDBSCAN, and $k$ ranges over a logarithmic grid. The primary metrics are AMI regret of the selected pipeline relative to the oracle, and the per-dataset Kendall's $\tau$ between each score and AMI over the pool.
The decisive check is the cross-encoder pool. If $T$'s regret is not significantly lower than UMAP-Silhouette's there, the proposal's central claim is refuted. A second decisive check is the structureless control: Gaussian noise and feature-shuffled versions of each dataset, where $T$ should be near zero while geometric indices may not be. A third check applies random anisotropic rescaling of the raw features, following the Maximin Separation Probability paper's protocol, to measure how much each score's ranking changes.
We will also report results by tier. On the fine-grained and far-OOD tiers, a stable partition by background or texture could score well while disagreeing with labels, as the semantic-gap critique in Evaluating Unsupervised Learning Results: Making the Seemingly Impossible Possible warns. We expect $T$ to share this weakness with every internal score and will measure how often its top pick is stable but semantically off.
The dense study repeats DSE's checkpoint protocol on COCO-Stuff, PASCAL VOC, ADE20k and Cityscapes, and compares Kendall's $\tau$ and selected mIoU against DSE's reported figures of about 0.57 and +3.0%.
References
- An Empirical Study into Clustering of Unseen Datasets with Self-Supervised Foundation Models
- The Need for Unsupervised Outlier Model Selection: A Review and Evaluation of Internal Evaluation Strategies
- Context Sensitive Information: Model Validation by Information Theory
- Maximin Separation Probability Clustering
- Zero-shot Clustering of Embeddings with Pretrained and Self-Supervised Learnt Encoders
- The DBCV index is more informative than DCSI, CDbw, and VIASCKDE indices for unsupervised clustering internal assessment of concave-shaped and density-based clusters
- Exploring Structural Degradation in Dense Representations for Self-supervised Learning
- Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation
- Evaluation of Unsupervised Learning Results: Making the Seemingly Impossible Possible