Deep Clustering Without Predefined Cluster Counts

21 concepts18 papers

Deep learning has revolutionized unsupervised representation learning, yet traditional deep clustering models suffer from a shared architectural dependency: they require the total number of clusters ($K$) to be specified as a fixed hyperparameter before training begins.

Whether partitioning images, text, or graphs, this fixed-$K$ constraint forces neural networks to map continuous data spaces into an arbitrarily rigid discrete cardinality. In real-world discovery scenarios, open-world environments, or domain transfer tasks, the true number of semantic categories is rarely known a priori. Overcoming this dependency requires shifting away from flat, fixed-target allocation toward architectures that can autonomously discover data granularity.

The Architectural Rigidity of Fixed-K Clustering

To understand why deep clustering models are traditionally locked into a predefined cluster count, we must look at their loss functions and final layers. Most deep clustering frameworks optimize representations by aligning latent features with an explicit set of target destinations.

For example, generative approaches like Variational Deep Embedding (VaDE), introduced by Jiang et al. in [1], replace the single-Gaussian prior of a standard Variational Autoencoder with a Gaussian Mixture Model (GMM). Similarly, Multi-Modal Deep Clustering (MMDC), proposed by Shiran et al. in [2], forces a network to map images directly to points sampled from a predefined GMM.

Fixed-K Paradigm (Rigid Architecture):
[Input Data] ──> [Deep Representation] ──> [Fixed Allocation Layer (K Outputs/Prototypes)]
                                                    │
                                                    └──> Forced to fit K groups, even if wrong.

In contrastive learning paradigms, the constraint manifests as a fixed set of cluster prototypes. Frameworks like ProPos, developed by Huang et al. in [3], and Center-Oriented Prototype Contrastive Clustering (CPCC), developed by Dong et al. in [4], optimize representations by pushing cluster prototypes apart while pulling instances closer to their assigned centers.

In all of these cases, the neural network requires a static output dimension or matrix size to compute gradients. A classification head needs exactly $K$ logits, and a prototype matrix needs exactly $K$ vectors.

As noted by Plant et al. in [5], if this user-defined estimate is incorrect, the optimization process is forced to distort or destroy the natural structural information within the data simply to satisfy the mathematical constraints of the fixed architecture. If $K$ is set too low, distinct semantic boundaries are collapsed; if $K$ is too high, coherent groups are artificially fragmented.

This limitation becomes highly problematic when models must generalize. As Zhao et al. demonstrate in [6], a deep clustering model optimized for a specific source domain frequently crumbles when transferred to a target domain where the underlying category count is dynamic or entirely different.

Mechanism 1: Divisive Hierarchical Trees

One prominent strategy for breaking the fixed-$K$ constraint is to replace flat, single-layer partitioning with top-down hierarchical splitting. Instead of forcing the network to organize data into $K$ groups simultaneously, these methods allow the network to discover the natural depth of the data incrementally.

Divisive Hierarchical Paradigm (Dynamic Architecture):
                 [Root Representation]
                        ╱    ╲
                       ╱      ╲
              [Cluster A]    [Cluster B]
                ╱    ╲           (Stop: Homogeneous)
               ╱      ╲
      [Cluster A1]  [Cluster A2]

In the Deep Embedded Cluster Tree (DeepECT) framework, Plant et al. implement a divisive hierarchical approach that simultaneously learns data representations and a cluster tree [5]. Rather than relying on a static classification layer, DeepECT utilizes a "projected compression loss." The model starts with the entire dataset as a single root node and recursively splits clusters into sub-clusters. Because the compression loss evaluates whether a split genuinely preserves structural information, the tree can grow dynamically and cease splitting when further divisions no longer yield meaningful compression.

A parallel mechanism is explored by de Mello et al. in their HC-MGAN framework [7]. They exploit the natural "specialization" tendency of Multi-Generator Generative Adversarial Networks (MGANs). By training a classifier to distinguish between different data generators and iteratively refining those splits top-down, the system organizes high-dimensional data into a semantically coherent tree.

Why it works: By framing clustering as a recursive tree-building process rather than a single flat assignment, the network avoids the need for a fixed output layer. The model can autonomously determine the optimal number of categories by evaluating local data distributions at each node, stopping the split whenever a cluster achieves structural homogeneity.

Mechanism 2: Differentiable Information Compression

An alternative to hierarchical splitting is to eliminate explicit centroids, prototypes, and mixture components altogether. Instead, the network can optimize a global, structural objective derived from information theory that naturally balances cluster complexity and data fidelity.

This principle is realized in Neuromap, a framework developed by Blöcker et al. for community detection in networks [8]. Neuromap bridges traditional network science and deep learning by converting the information-theoretic "map equation" into a fully differentiable loss function optimized by Graph Neural Networks (GNNs).

Information-Theoretic Optimization:
Loss = Under-clustering Penalty (High Entropy) + Over-clustering Penalty (High Codebook Overhead)
                          │
                          ▼
           [Spontaneous Convergence to Optimal K]

Why it works: The map equation measures how efficiently a random walk across a data graph can be compressed using a two-level codebook (analogous to how city names and street names organize geographic space).

  • If the network assigns all data points to a single massive cluster (under-clustering), the description lacks precision, resulting in a high entropy penalty.
  • If the network assigns every data point to its own unique cluster (over-clustering), the index codebook becomes bloated, resulting in a high overhead penalty.

By minimizing this compression metric directly via gradient descent, the GNN naturally converges on the optimal number of clusters. The objective function itself acts as a dynamic scale, eliminating the need for human intervention or an a priori fixed-cardinality constraint.

Summary

The transition away from predefined cluster counts represents a fundamental shift in unsupervised deep learning. While early deep clustering methods relied on rigid architectural targets like fixed GMMs or static prototype matrices, modern approaches bypass these constraints. By leveraging recursive divisive hierarchies or differentiable information-theoretic compression losses, contemporary models allow the underlying structure of the data to dictate the network's capacity, rather than forcing the data to conform to a rigid architectural bottleneck.

References

[[1]] Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
Jiang et al., IJCAI 2017.

[[2]] Multi-Modal Deep Clustering: Unsupervised Partitioning of Images
Shiran et al., arXiv 2019.

[[3]] Exploring Non-Contrastive Representation Learning for Deep Clustering
Huang et al., arXiv 2021.

[[4]] Center-Oriented Prototype Contrastive Clustering
Dong et al., arXiv 2025.

[[5]] DeepECT: The Deep Embedded Cluster Tree
Plant et al., Data Science and Engineering 2020.

[[6]] Transferable Deep Clustering Model
Zhao et al., arXiv 2023.

[[7]] Top-Down Deep Clustering with Multi-Generator GANs
de Mello et al., AAAI 2022.

[[8]] The Map Equation Goes Neural: Mapping Network Flows with Graph Neural Networks
Blöcker et al., NeurIPS 2024.

Key Foundational Work

Multi-Modal Deep Clustering: Unsupervised Partitioning of Images (2019) — contributed 1 concept elements to this direction.
Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering (2017) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 18 papers
  2. Version 1 · 2026-10-04 · 40 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_predefined_cluster_count_dependency_4c37efaf,
  title = {predefined cluster count dependency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predefined-cluster-count-dependency/rdr_4c37efafdf464b6280b74038ef637d4d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_predefined_cluster_count_dependency_4c37efaf_v2,
  title = {predefined cluster count dependency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predefined-cluster-count-dependency/rdr_4c37efafdf464b6280b74038ef637d4d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake