Abstract
Deep Subspace Clustering Networks (DSC) provide an efficient solution to the problem of unsupervised subspace clustering by using an undercomplete deep auto-encoder with a fully-connected layer to exploit the self expressiveness property. This method uses undercomplete representations of the input data which makes it not so robust and more dependent on pre-training. To overcome this, we propose a simple yet efficient alternative method - Overcomplete Deep Subspace Clustering Networks (ODSC) where we use overcomplete representations for subspace clustering. In our proposed method, we fuse the features from both undercomplete and overcomplete auto-encoder networks before passing them through the self-expressive layer thus enabling us to extract a more meaningful and robust representation of the input data for clustering. Experimental results on four benchmark datasets show the effectiveness of the proposed method over DSC and other clustering methods in terms of clustering error. Our method is also not as dependent as DSC is on where pre-training should be stopped to get the best performance and is also more robust to noise.
Introduction
Subspace clustering is the task of grouping data points into clusters based on the assumption that they lie in low-dimensional subspaces. For years, the gold standard has been Deep Subspace Clustering (DSC), which uses an autoencoder to compress an image into a tiny "latent" representation before clustering. However, DSC suffers from two major weaknesses: fragility to noise and extreme sensitivity to pre-training duration. Because the encoder aggressively compresses data, it often loses fine details, making the model perform poorly if the input images are even slightly noisy.
Methodology: The Overcomplete Approach
The proposed Overcomplete Deep Subspace Clustering Networks (ODSC) rethink the encoder architecture. Instead of just using a standard "undercomplete" encoder (which reduces spatial dimensions via max-pooling), ODSC runs a second overcomplete encoder branch in parallel.
In the overcomplete branch, the network uses upsampling instead of pooling. This forces the deeper layers to represent the data in a higher spatial dimension than the input. While standard networks increase their receptive field to capture the "big picture," the overcomplete branch's receptive field actually shrinks in deeper layers, allowing the network to capture extremely fine details and local structures. The model fuses these overcomplete features with standard undercomplete features before passing them to a self-expressive layer to learn the affinity matrix.
Key Insights: Finer is Better
The core "aha moment" of this research is that while compression is good for abstraction, expansion is better for detail. The overcomplete branch captures edges and textures that the undercomplete branch typically ignores.
By having two different "views" of the data (coarse and fine), the model becomes much less sensitive to the exact moment pre-training is stopped. Furthermore, overcomplete representations are inherently more robust to noise. When 50% noise was added to MNIST, DSC’s error increased significantly, while ODSC’s error remained remarkably stable with only a ~1% increase.
Experimental Results
ODSC consistently outperformed DSC and other traditional methods (like SSC and LRR) across four classic computer vision benchmarks: MNIST, ORL, COIL20, and Extended Yale B.
- COIL20: Reduced clustering error from 5.14% (DSC) to 2.5%.
- MNIST: Reduced error from 25.0% (DSC) to 18.8%.
- Extended Yale B: Achieved a low error of 2.22%.
Despite having two encoder branches, the authors carefully reduced the number of filters in each layer so that the total parameter count remains almost identical to the original DSC model.
Conclusion
ODSC demonstrates that overcomplete representations can significantly enhance the robustness and accuracy of deep subspace clustering. While the experiments were conducted on relatively small datasets, the results suggest that maintaining high-resolution features is critical for unsupervised tasks. Future work may explore scaling this approach to larger, uncurated datasets and investigating whether overcomplete decoders could further improve reconstruction and clustering performance.
Summary updated