Abstract
Image clustering remains a significant challenge due to the high dimensionality of visual data. While deep learning approaches like Deep Embedded Clustering (DEC) have improved performance by learning latent representations, they often rely on flat clustering algorithms like k-means that fail to capture hierarchical relationships. This paper introduces two innovations: CAE-MLE, which integrates Agglomerative Clustering into a Convolutional Autoencoder, and Deep Inverse Feature Learning (Deep IFL), which utilizes clustering residuals as informative features. Experimental results on MNIST and USPS datasets demonstrate that this approach significantly outperforms existing deep clustering methods, achieving state-of-the-art accuracy of 97.50% on MNIST.
Introduction
Clustering high-dimensional data such as images is a fundamental unsupervised learning task. Recent advances have focused on Deep Clustering, where a neural network learns a low-dimensional embedding optimized for a specific clustering objective. However, dominant models like Deep Convolutional Embedding Clustering (DCEC) typically assume a flat cluster structure.
The authors of this paper argue that images possess inherent hierarchical sub-patterns—for instance, the digits '6' and '9' share similar curves. By ignoring this hierarchy, flat models lose discriminative power. Additionally, they propose that the "error" or relationship between a sample and various cluster centroids is not just a loss signal but a rich source of feature information that can be harvested to improve final clustering performance.
Methodology
The proposed framework consists of two primary components designed to work in tandem: hierarchical latent learning and inverse feature extraction.
CAE-MLE: Hierarchical Latent Learning
The Convolutional Autoencoder with Multi-Level Embedding (CAE-MLE) architecture utilizes a standard encoder-decoder structure to learn compressed representations. Unlike previous models that apply k-means to the latent space, CAE-MLE employs Agglomerative Clustering (AC) with Ward linkage. AC builds a nested hierarchy by iteratively merging clusters based on affinity, allowing the model to learn a latent space that reflects the multi-level patterns of the input data.
Deep Inverse Feature Learning (Deep IFL)
Deep IFL is a meta-strategy that treats clustering "errors" as high-level features. Using an "Inner Folding" process (partitioning data into training and held-out sets), the model extracts several metrics for each instance:
- Confidence: The ratio of elements in the nearest cluster to total instances.
- Weight: The vector of distances from the instance to all cluster centroids.
- Proximity: The distance to the single nearest centroid.
These "inverse features" are concatenated with the original latent features, providing a more robust representation for the final clustering stage.
Experimental Results
The models were evaluated on the MNIST and USPS handwritten digit datasets. MNIST contains 70,000 images (28x28), while USPS contains 9,298 images (16x16). These datasets are standard benchmarks for unsupervised representation learning.
As shown in Figure 5, the proposed methods significantly outperform a wide range of baseline and state-of-the-art clustering models. CAE-MLE alone improved upon the standard DCEC baseline by 7% to 14%. When combined with Deep IFL, the performance reached a peak Accuracy (ACC) of 97.50% and a Normalized Mutual Information (NMI) of 93.09% on MNIST.
On the USPS dataset, the model achieved 96.89% ACC, a massive improvement over the 79.00% achieved by standard DCEC and the 72.00% of specialized hierarchical models like DeepECT.
Conclusion
By moving beyond flat clustering assumptions and repurposing clustering residuals as features, the authors demonstrate that deterministic autoencoders can achieve or exceed the performance of more complex generative models like GANs and VAEs. While the computational cost of Agglomerative Clustering poses a challenge for massive datasets, the significant accuracy gains on standard benchmarks validate the importance of hierarchical and inverse feature learning in deep clustering.
Summary updated