Abstract
Deep Subspace Clustering Networks (DSC) provide an efficient solution to the problem of unsupervised subspace clustering by using an undercomplete deep auto-encoder with a fully-connected layer to exploit the self expressiveness property. This method uses undercomplete representations of the input data which makes it not so robust and more dependent on pre-training. To overcome this, we propose a simple yet efficient alternative method—Overcomplete Deep Subspace Clustering Networks (ODSC) where we use overcomplete representations for subspace clustering. In our proposed method, we fuse the features from both undercomplete and overcomplete autoencoder networks before passing them through the self-expressive layer, thus enabling us to extract a more meaningful and robust representation of the input data for clustering. Experimental results on four benchmark datasets show the effectiveness of the proposed method over DSC and other clustering methods in terms of clustering error. Our method is also not as dependent as DSC is on where pre-training should be stopped to get the best performance and is also more robust to noise.
Introduction
Subspace clustering is a fundamental unsupervised learning task aimed at grouping data points into their respective categories without the aid of labels. The current state-of-the-art approach, Deep Subspace Clustering (DSC), utilizes a convolutional autoencoder to learn non-linear mappings. However, DSC suffers from two significant drawbacks: high sensitivity to noise due to its "undercomplete" (bottleneck) architecture, and instability during pre-training, where performance varies wildly based on the exact epoch at which reconstruction training is halted.
Methodology: Overcomplete Architectures
The proposed Overcomplete Deep Subspace Clustering (ODSC) network addresses these limitations by introducing an Overcomplete Branch alongside the traditional undercomplete one.
Dual-Branch Encoder
Instead of a single bottleneck, ODSC employs two parallel encoders:
- Undercomplete Branch: Uses standard convolution and max-pooling to capture abstract, high-level features by increasing the receptive field.
- Overcomplete Branch: Employs convolution followed by upsampling. This increases the spatial dimensionality of feature maps, constraining the receptive field and forcing the network to focus on fine-grained structural details like edges and textures.
Feature Fusion and Clustering
Latent representations from both branches are fused and passed into a "self-expressive layer." This layer learns an affinity matrix representing data point relationships, which is subsequently used for spectral clustering. A shared decoder ensures that the learned features remain grounded in the original data structure, maintaining parameter efficiency.
Experimental Results
The authors evaluated ODSC on four computer vision benchmarks: MNIST, ORL, COIL20, and Extended Yale B.
Performance and Robustness
ODSC demonstrated substantial improvements in clustering accuracy. For instance, on the COIL20 dataset, the clustering error dropped from 14.86% (DSC) to 2.5%. On MNIST, the error was reduced from 25% to 18.8%.
As shown in the feature visualizations above, the overcomplete branch (b) preserves much more structural detail compared to the coarse representations in the undercomplete branch (a). This preservation of detail translates directly to noise robustness; when 50% random noise was added to MNIST, ODSC's error increased by only ~1%, compared to over 3% for DSC.
Training Stability
A key advantage of ODSC is its reduced sensitivity to pre-training duration. While DSC performance fluctuates significantly depending on the stopping epoch, the fusion of overcomplete features acts as a stabilizer, providing a consistent signal for the clustering layer and resulting in much lower variance across different training runs.
Conclusion
ODSC proves that "compression" is not always optimal for unsupervised tasks. By preserving spatial resolution through overcomplete representations, the network captures structural nuances that bottlenecks discard. While future work could explore more complex fusion strategies or evaluate performance on larger "in-the-wild" datasets, ODSC establishes a more robust and stable framework for deep subspace clustering.
Summary updated