Abstract: A promising direction in deep learning research consists in learning representations and simultaneously discovering cluster structure in unlabeled data by optimizing a discriminative loss function. As opposed to supervised deep learning, this line of research is in its infancy, and how to design and optimize suitable loss functions to train deep neural networks for clustering is still an open question. Our contribution to this emerging field is a new deep clustering network that leverages the discriminative power of information-theoretic divergence measures, which have been shown to be effective in traditional clustering. We propose a novel loss function that incorporates geometric regularization constraints, thus avoiding degenerate structures of the resulting clustering partition. Experiments on synthetic benchmarks and real datasets show that the proposed network achieves competitive performance with respect to other state-of-the-art methods, scales well to large datasets, and does not require pre-training steps.
Introduction
Traditional clustering algorithms, such as $k$-means, often struggle with high-dimensional data like images because they rely on simple Euclidean distance metrics in the raw input space. While deep learning offers a solution by learning hierarchical representations, most existing "deep clustering" methods are fragmented. For instance, popular approaches like Deep Embedded Clustering (DEC) require a two-stage process: first training an autoencoder to learn features, and then fine-tuning for clustering.
Deep Divergence-based Clustering (DDC) addresses this gap by providing a principled, information-theoretic framework that performs feature learning and clustering simultaneously in a single end-to-end pipeline. By optimizing a novel loss function based on Cauchy-Schwarz divergence and geometric constraints, DDC discovers cluster structures from scratch without the need for cumbersome pre-training steps.
The DDC Framework
The core of the DDC approach is a flexible architecture—typically a convolutional neural network (CNN) for images or a multi-layer perceptron (MLP) for vectorial data—optimized via a three-part loss function designed to shape hidden representations and cluster assignments.
The DDC Loss Function
The loss function consists of three distinct terms:
- Cauchy-Schwarz (CS) Divergence ($l_1$): This term measures the dissimilarity between the probability density functions (PDFs) of the clusters. By maximizing this divergence, the network ensures that clusters are well-separated and internally compact. A Parzen window estimator is used to make this calculation data-driven and differentiable.
- Orthogonality Constraint ($l_2$): To prevent "degenerate solutions" where the network might collapse all data into a single cluster, this term encourages the cluster assignment vectors to be orthogonal. This acts as a regularizer promoting a balanced distribution of data across classes.
- Simplex Corner Constraint ($l_3$): Since the final layer uses a softmax activation, the output space is a simplex. This term pushes data points toward the "corners" of the simplex, resulting in "crisp" assignments where a point clearly belongs to one cluster rather than an ambiguous soft assignment.
Experimental Evaluation
The authors evaluated DDC on several benchmarks, including MNIST (70,000 handwritten digits), SEALS (aerial RGB images), and Reuters (800,000 news stories). Unlike many competitors, DDC does not require a "warm start" from a pre-trained autoencoder and can learn features and clusters simultaneously from random initialization.
Key Results
DDC and its ensemble version (DDC-VOTE) consistently outperformed baselines like $k$-means, Spectral Embedded Clustering (SEC), and DEC:
- MNIST: DDC-VOTE achieved an unsupervised accuracy of 88.49%, significantly higher than DEC (84.31%) and $k$-means (53.33%).
- Reuters: DDC-VOTE reached 77.62% accuracy, outperforming DEC at 63.79%.
- Scalability: Because the method uses mini-batch training, it avoids the $O(n^2)$ memory complexity that plagues spectral clustering, making it viable for massive datasets.
Discussion and Conclusion
DDC demonstrates that end-to-end deep clustering is not only possible but highly effective. By explicitly exploiting the geometry of the softmax output (the simplex) and adapting Cauchy-Schwarz divergence into a differentiable loss, the model makes confident and distinct clustering decisions.
However, some limitations remain. The $l_2$ orthogonality term inherently encourages clusters of similar sizes, which may struggle with highly imbalanced real-world data. Additionally, like many unsupervised tasks, DDC can be sensitive to local minima, often requiring multiple runs or voting schemes to ensure stability. Future work may explore alternative divergence measures and extend the framework to multi-view clustering, where data is sourced from multiple modalities simultaneously.
Summary updated